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A Message from 
the Auditor General of Canada 


I am pleased to present this first Status Report to the House of Commons. It 
reports on our audits of the steps that federal government departments and 
agencies have taken to implement some of the recommendations we have 
made within the past five years. Taking into account Parliament’s response to 
_ this report, I intend to present a status report to the House of Commons each 
_ year, with the next one in the spring of 2003. 


Sheila Fraser, FCA , 

Auditor General of Canada A new approach to following up on past reports to Parliament 

Although follow-up on our previous recommendations has been part of our 
regular work for many years, this year we have taken a new approach to the 
way we carry out our follow-up. This Status Report is a new approach to 
reporting on that work. 


What distinguishes the five chapters in this report from our previous follow- 
up work is the way we selected the issues, the scope and depth of our review, 
and the assurance our findings provide. 


From routinely reporting on all previous recommendations two years after the 
original audit, with these new reports we move to a focus on the issues that 
are most significant—those that are systemic, timely, still relevant, carry 
higher risk, and in our view are of interest to parliamentarians. In some cases, 
we extended our work to include new issues that are relevant to the original 
audit work. 


Our previous approach relied on status reports from departments on action 
they had taken to address our recommendations; we would evaluate those 
reports to assess whether they seemed plausible. In some cases we would do 
additional work where we believed it was warranted. The chapters in this 
report present instead the results of conventional audits, meeting all our 
normal standards of evidence and quality management. 


We also decided to change the way we report our follow-up work. Previously, 
we included follow-up work in reports with chapters devoted to new audit 
work. We realized that the new recommendations overshadowed the impact 
our follow-up work had on both the departments’ priorities for action and the 
visibility of the issues. The lack of visibility tended to conceal successful 
corrective action as well as failures to act. Now, the yearly Status Report will 
be devoted entirely to our follow-up work and will become one of the four 
reports I provide each year to the House of Commons. 


Part of a continuing evolution of service to Parliament 


Our new and enhanced approach to following up on previous 
recommendations is consistent with the evolution of this Office’s efforts to 
provide Parliament with the information it needs to determine whether or not 
Canadians are getting value for money. 
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It so happens that the publication of this report coincides with the 25th 
anniversary of the Auditor General Act of 1977. That legislation formally 
broadened the Auditor General’s mandate to include reporting on whether 
government policies are being implemented economically, efficiently, and 
with adequate means for judging their effectiveness. The Act marked the 
birth of value-for-money auditing in Canada—that is, examining whether 
Canadians are getting their money’s worth for their tax dollars. 


Legislative auditing has changed a great deal since the appointment of the 
first Auditor General of Canada in 1878. As government has evolved and 
become more complex, the kinds of information Parliament needs to hold the 
government to account have also changed. Accordingly, my predecessors 
adapted their approach to auditing, with Parliament's encouragement, to 
provide the needed information. 


The roots of value-for-money auditing in Canada date back more than 50 
years to the Financial Administration Act of 1951, which empowered the 
Auditor General to report to Parliament “any other case that he considers 
should be brought to the attention of the House.” The fifth Auditor General, 
Watson Sellars, started in 1958 to report what he called “unproductive 
expenditures” of government funds. His successor, Maxwell Henderson, 
reported three such examples in his first report of 1960, and asked Parliament 
whether or not he should continue. The Public Accounts Committee said 
yes. 


Value -for-money auditing was formally launched during James Macdonell’s 
term as Auditor General, when the Auditor General Act of 1977 expanded his 
mandate to include reporting on whether government policies were being 
implemented economically and efficiently and whether departments had in 
place reliable, up-to-date measures of effectiveness. Macdonell was the first 
Auditor General to make recommendations to departments on how to 
remedy problems rather than simply enumerating them for Parliament and 
was the first to publish departments’ reactions to findings and 
recommendations in his reports. Both innovations were carried on by his 
successors and are incorporated in the new Status Report. 


Our reports do lead to change 


Since 1977, the Auditor General has submitted to the House of Commons 
hundreds of value-for-money chapters with thousands of recommendations. 
Public interest peaks immediately following tabling, when the news is fresh 
and the media are full of stories that shock, amuse, or merely perplex. Like my 
predecessors, I note that public attention to the issues has not always 
corresponded with their relative significance. 


Members of Parliament focus most visibly on the report during Question 
Period immediately following tabling. A less visible but more thorough 
examination of the issues we raise occurs in the following months, when 
various committees of the House and the Senate, and particularly the 
Standing Committee on Public Accounts, consider our reports in detail and 
invite senior officials from the departments concerned to answer committee 
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members’ questions. Following these discussions, the Public Accounts 
Committee makes its own report to the House with recommendations to the 
government. Other standing committees of the House of Commons or the 
Senate also review our reports and issue their own. 


The Public Accounts Committee’s consideration of issues raised in our 
reports and its questioning of senior officials of departments about their 
actions to address them are not always noticed by the public. Though the 
Committee’s meetings are usually open to the public, few journalists attend 
and there is little news coverage of the hearings, the Committee’s reports, or 
the government’s responses. As a result, the public may believe that after the 
initial impact of our reports, departments take no action. 


However, that impression is not supported by our annual compilation of the 
status of recommendations we have made over the five preceding years and 
our assessment of departments’ progress. We collect information on all 
recommendations for five years after an audit, and we track in a database 
what departments say they have done to address them. In our annual 
performance report, we publish the percentage of our recommendations that 
have been implemented satisfactorily. For the period 1996-2000, for example, 
our database shows that departments reported having completed the 
recommended action on a quarter of the recommendations we made and 
having made satisfactory progress on half of them. 


At first, I was surprised to find that the satisfactory level of implementation 
indicated in our database was not matched by our evaluations of departments’ 
progress in the five chapters of this Status Report—which are the results of 
our new approach to follow-up, one that is more rigorous and more intensive 
than the assessments in our database. 


I can think of at least two reasons for this. 


First, audit recommendations by their nature are intended to correct 
problems. Follow-up on past recommendations, then, means looking at areas 
where there were problems—some of which will need more time and 
resources to resolve. Since our new Status Report focusses on the issues that 
we believe need to be brought once again to Parliament’s attention—the 
most complex and most significant issues that pose the greatest risk—it could 
be expected that our findings would be less than satisfactory. 


In addition, some of the problems touched on in these five chapters are long- 
standing, extremely complex issues such as federal support for health care. In 
such areas, the federal government is not the only player, and we can 
comment only on how it carries out its share of the responsibility. However, in 
the following chapters the unsatisfactory progress we report indicates that the 
government needs to step up its efforts to correct problems that we have 
identified and that Parliament—and in most cases the departments 
themselves—agree must be addressed. 
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Listening to parliamentarians 


Members of Parliament, particularly members of the Standing Committee on 
Public Accounts, have told us they want a more thorough follow-up of 
departments’ progress in implementing our recommendations on key issues. 
Together, this Status Report and our annual monitoring of action on all our 
recommendations provide an overview of the government’s progress in 
addressing problems identified by our audits. 


I hope parliamentarians find that this new report provides the information 
they need to hold the government accountable and is a fitting way to mark 
the 25th anniversary of the Auditor General Act and the introduction of 
value-for-money auditing. 


Sleeon gear 


Sheila Fraser, FCA 
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Human Resources Development Canada __ 


The Integrity of the Social Insurance Number 


The audit work reported in this chapter was conducted in accordance with the legislative mandate, policies, and practices of the 
Office of the Auditor General of Canada. These policies and practices embrace the standards recommended by the Canadian 
Institute of Chartered Accountants. 
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Human Resources Development Canada 
The Integrity of the Social Insurance Number 


Assistant Auditor General: Maria Barrados 
Principal: Peter Simeoni 


Key Message 


11 Human Resources Development Canada (HRDC) has been slow to address many of the observations and 
recommendations arising from our 1998 audit of the management of the Social Insurance Number (SIN). In 2000 
we reported that some improvements had been made in SIN records and the number of SIN-related fraud 
investigations had increased. In 2002 we returned, expecting to find that the problems reported in 1998 would have 
been largely resolved. Instead, we found that progress on some key issues has been limited. The continuing weakness 
in the issuing of SINs leads us to conclude that HRDC has not done enough to safeguard and strengthen the 
integrity of the SIN. : 


ORIGINAL ISSUES 


1.2. The SIN had become a de facto national identifier, 
contrary to the government’s intent. We encouraged the 
government to consider using planned privacy legislation 
to address concerns about the use of SIN outside the 
federal government. 


PROGRESS 


The government reaffirmed its policy that the SIN was 
only an account number for authorized federal programs. 
To help protect personal information (including the SIN) 
outside the federal government, it is implementing the 
Personal Information Protection and Electronic 
Documents Act. HRDC has examined the use of the SIN 
in the private sector and has made efforts to raise public 
awareness of how the SIN should be used. 


1.3 HRDC needed to strengthen its process for issuing 
a SIN. It also needed to address the risks associated 
with SINs that do not expire and are issued to people 
such as visitors to Canada, who often need them only 
temporarily (900 series). 


HRDC has made little change since 1998 in the way it 
issues SINs—regular ones or 900-series. In our view, the 
policies and practices of the Department do not meet the 
intent of the Employment Insurance Act and Regulations 
because it is not doing enough to identify SIN applicants 
properly. 


The Department has been providing better work tools to 
staff, albeit slowly. It recently completed a study of the 
way it issues SINs, as a basis for strengthening the 
integrity of the process. 


1.4 HRDC needed to bring the reliability and 
completeness of its database of SIN records to an 
appropriate level. 


Millions of SIN records have been cross-checked with 
other HRDC programs, although miilions more will not 
be checked. The number of usable SINs for people over 
20 years old exceeds the Statistics Canada Census figure 
by 5 million. HRDC believes there is little risk that 

2.6 million of these SINs will be used as they have been 
dormant for some time. It recently started investigating 
activity on dormant SIN records. These SINs can still be 
used as a first step in accessing other federal programs. 


1.5 HRDC needed to strengthen its approach to fraud 
investigations involving the SIN. 


HRDC has increased the number of investigations but 
they are not guided by a comprehensive review of where 


the risks lie in issuing SINs. However, the Department 
has started providing better tools and training to its staff. 


RATING* 


SATISFACTORY. 
PROGRESS 


LIMITED 
PROGRESS 


LIMITED 
PROGRESS 


LIMITED 
PROGRESS 


*Possible ratings are completed, satisfactory progress, limited progress, no progress, rejected, unknown. (See About the Audit for an explanation of the ratings.) 
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Human Resources Development Canada has responded. HRDC agrees with our recommendations and has 
committed itself to specific actions. HRDC will seek authority to cancel any SINs that have not been used in five 
years. It will also seek authority to place expiry dates on the SIN cards of people who are neither Canadian citizens 
nor permanent residents and will no longer accept photocopies of identity documents as proof of identity. 


The Treasury Board Secretariat has responded. The Secretariat agrees with the recommendation we addressed to it. 
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Introduction 


16 In 1998 we audited the management of the Social Insurance 
Number (SIN). We reported several problems in the way it was managed. 
We conducted a follow-up in 2000. Based on more recent audit work, this 
chapter provides a status report on the government’s efforts to improve the 
integrity of the SIN. 


What is the Social Insurance Number? 


1.7. The Social Insurance Number was created in 1964 as a file identifier 
for the Canada Pension Plan, Quebec Pension Plan, and employment 
insurance (then unemployment insurance) programs. Under the current 
Employment Insurance Act, the Canada Employment Insurance Commission 
assigns “a number that is suitable for use as a file number or account number 
or for data processing purposes and the number so assigned...is the person’s 
Social Insurance Numbet....” The SIN is a unique, nine-digit number 
assigned to each individual, and a person legitimately has only one number at 
a time. 


18 The government’s current policy views the SIN as a file identifier or 
account number for specifically authorized activities (Exhibit 1.1). The 
Employment Insurance Act and Regulations require that a SIN be issued only 
after an applicant's identity and citizenship status have been determined. 


Who is involved in administering the SIN? 


19 The Canada Employment Insurance Commission is named in the Act 
and Regulations as the body responsible for SIN administration. On behalf of 
the Commission, Human Resources Development Canada (HRDC) issues 
SINs and maintains the Social Insurance Register, which holds all relevant 
information on SIN applicants. The Treasury Board is responsible for the SIN 
policy, and the Treasury Board Secretariat is responsible for developing 
guidelines that govern how federal departments collect and use the SIN. The 
Canada Customs and Revenue Agency uses the SIN in administering the 
Income Tax Act. The Office of the Privacy Commissioner of Canada monitors 
federal organizations for their compliance with the Privacy Act, which governs 
the protection of personal information, including the SIN. For example, 
departments are required to inform the Privacy Commissioner of any 
significant changes to existing uses of personal information, specifically any 
new uses or disclosures. 


1.10 In 2001-02, HRDC spent about $16.7 million on SIN administration 
and collected close to $2 million from people applying for replacement SIN 
cards. It issued a total of 1.2 million new cards and replacements. 


Why is the integrity of the SIN important? 


111 The SIN is needed to access federal programs. Several of the federal 
government’s largest programs use the Social Insurance Number. In 2000-01 
the SIN was the key account number on about three million applications for 
Employment Insurance benefits that totalled $9.5 billion and Canada Pension 
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Plan and Old Age Security benefits that totalled $43.7 billion. Applicants for 
benefits under these programs need a SIN. In 1998 we reported that by using 
the SIN, various programs had been able to avoid or recover overpayments 
and improve administrative efficiency, thereby achieving significant savings. 


1.12 Widespread use of the SIN outside the federal government. Use of 
the SIN in the private sector began when the government made it a personal 
identifier for income tax, in 1967. Since then, the Income Tax Act has been 
amended several times to require that various public agencies collect a 
person’s SIN in their transactions and to ask the same of private sector 
organizations, such as insurance companies and financial institutions. Various 


Exhibit 1.1 Uses of the SIN in the public and private sectors 


Public sector 


Federal government—Areas of use authorized by Treasury Board policy 


Human Resources Development Canada Other legislation and programs 


Canada Education Savings Grants 
Canada Labour Code 
Canada Pension Plan 


Canada Student Financial Assistance 
Act and Regulations 


Canada Student Loans Act 
Employment Insurance Act 
Labour Adjustment Benefits Act 


Canada Elections Act 
Canada Wheat Board Act 


Family Orders and Agreements 
Enforcement Assistance Act 


Farm Income Protection Act 


Immigration Resettlement Assistance 
Program 


Income and Health Care programs 


(Veterans Affairs Canada) 

National Dose Registry for occupational 
exposures to radiation 

Rural and Native Housing Program 
(Canada Mortgage and Housing 
Corporation) 

Social Assistance and Economic 
Development Program (Indian and 
Income Tax Act Northern Affairs Canada) 


ae tax appeals and adverse Veterans Allowance Regulations 
ecisions 


Tax Rebate Discounting Regulations 


Labour Adjustment Review Board 
Old Age Security Regulations 


Canada Customs and Revenue Agency 


Excise Tax Act (Part IX—GST) 


Gasoline and Aviation Gasoline Excise 
Tax Application Regulations 


Other jurisdictions—some areas of use 


Health insurance 
Social assistance 
Welfare 

Workers’ compensation 


Private sector 


Interest reporting to the Canada Customs and Revenue Agency (by, for example, banks, 
trust companies, stock brokers) 


Income tax identification 
Employer file number 
Client identification (for example, membership in video stores) 


Credit check (by, for example, car dealerships, department stores, cellular telephone 
companies, landlords) 


Source: Statistics Canada and Treasury Board Secretariat 
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documents we examined also indicated that provinces and municipalities use 
the SIN in administering social assistance and other programs. In short, many 
organizations collect SINs from Canadians and then retain that information. 


113 The widespread use of the SIN for many different purposes gives it a 
value that means it must be linked with only its rightful owner. When it is 

not, government benefits, tax refunds, or bank credit may go to the wrong 
person. 


1.14 Opportunities for SIN fraud. According to HRDC, between 1964 and 
1976 almost 19 million SINs were issued without requiring the applicants to 
provide proof of their identity. In 1976, the government began requiring that 
new applicants and card holders seeking replacement cards furnish 
documents such as birth certificates to prove their identity. Due to cost 
considerations, the government decided not to require the 19 million people 
who already had a SIN to register again with proof of their identity. 


1.15 According to HRDC, the proof-of-identity requirements established in 
1976 are no longer sufficient. New technologies have made it relatively easy 
to forge identity documents such as birth certificates. HRDC has anecdotal 
reports from the Royal Canadian Mounted Police, other police agencies, and 
its own employment insurance investigators that illustrate how easily fraud 
can be perpetrated and SINs issued that should not be issued. In addition, the 
plastic SIN card can be used as an identity document, although it has no 
modern security features and jis easily forged. There is a wide range of fake 
documents available on the Internet, including birth certificates. The formula 
for generating a false SIN that will look like a valid one can also be accessed 
on the Internet. 


116 Impact of SIN fraud. In 1998 we noted that obtaining a SIN 
fraudulently can be part of developing a false identity and an initial step in 
schemes to obtain government services or credit fraudulently or evade taxes. 


1.17 In both Canada and the United States, identity fraud is a growing 
concern. The SIN and its U.S. counterpart, the social security number, can 
play an important role in this kind of crime. American authorities report that 
social security numbers, along with dates of birth and mothers’ maiden 
names, are among personal identifiers often used in a wide range of identity- 
related frauds. These frauds can be steps toward far more serious crimes. 


1.18 The Canadian Council of Better Business Bureaus reports that stolen 
identity is one of its top three concerns. Equifax Canada Inc., a consumer and 
commercial credit reporting company, states that the incidents of identity 
theft it handles increased from 4,000 cases in 1998 to over 12,000 cases in 
2001. The U.S. Federal Trade Commission reported that in 2001 it processed 
more than 86,000 reports from victims of identity theft. In a recent report to 
Congress, the General Accounting Office concluded that the prevalence and 
cost of identity theft in the United States appear to be growing. 


119 According to HRDC, the expanded use of the SIN by other levels of 
government and institutions has both increased the potential for SIN fraud 
and extended its impact. For example, once someone has established a false 
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identity that includes a SIN obtained fraudulently from HRDC, that identity 
can be used to access federal and provincial social programs, to defraud 
banks, and to misrepresent income to the Canada Customs and Revenue 
Agency. As the Web site of the Office of the Privacy Commissioner of Canada 
notes, a valid SIN can also be used to steal a real identity. It is possible to use 
someone else’s SIN along with other personal information to apply for a credit 
card, open a bank account, and rent vehicles, equipment, or accommodation 
in that person’s name, leaving the victim responsible for the bills, charges, bad 
cheques, and taxes. Exhibit 1.2 provides recent examples of the kinds of 
crimes in which the SIN has played a part. 


Exhibit 1.2 Crimes involving the Social Insurance Number 


While some of these issues are outside the direct control of the federal government, 
crimes involving SINs are a problem. 


¢ An individual arriving in Canada applied for refugee status, which was denied. 
He began working illegally and subsequently applied for Employment Insurance 
benefits using his son’s SIN card. The individual’s common-law wife was also 
working illegally, using their daughter’s SIN; his brother was committing the same 
crime, using his own son’s SIN. 


An individual was caught making false documents and using them to impersonate 
individuals. At the time of his arrest, he had in his possession various fraudulent 
cards, including SIN cards with various names. With the false documents, he had 
been able to change the victims’ personal identification numbers for access to their 
lines of credit, credit card accounts, and chequing and savings accounts. Using 
similar methods, he succeeded in obtaining thousands of dollars from two victims’ 
bank accounts. 


Source: Human Resources Development Canada 


What we found in 1998 


1.20 Our 1998 audit found that the SIN had become a common numerical 
identifier both inside and outside the federal government in a wide range of 
income-related transactions and benefits. We reported several weaknesses in 
the administration of the SIN, including the following: 


¢ Information held by HRDC on SIN holders, particularly on births and 
deaths, was not always complete and accurate. 


* Procedures to apply for a SIN were not sufficient to guard against fraud 
and abuse. Millions of SIN holders who had registered before the proof- 
of-identity program was introduced in 1976 had not been asked to 
furnish documents that proved their identity. This exacerbated the risk 
of misrepresentation. 


* The provinces and financial institutions were required by the Income Tax 
Act and Regulations to collect SINs for tax purposes, but could not 
validate the SIN numbers provided by their clients. 
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* SIN errors, abuse, and misuse affected many federal and provincial 
programs and the private sector. Collectively, the impact may have been 
sizable. 


* HRDC was dedicating minimal effort to investigating SIN fraud and 
abuse, and penalties were minimal, with no real impact on deterrence. 


* Unregulated use in the private sector (except in Quebec) made the SIN 
vulnerable to abuse—both fraud and invasion of privacy. 


1.21 Our report contended that it was time to review the current role, 
objectives, and uses of the SIN in light of the important function it serves in 
government and society. The government needed to state clearly the level of 
integrity and protection of privacy expected in the system. 


What has happened since 1998? 


1.22. HRDC, the Canada Customs and Revenue Agency (CCRA), 
Citizenship and Immigration Canada, Industry Canada, and the Department 
of Justice Canada accepted our 1998 recommendations and observations. 
The Standing Committee on Public Accounts and the Standing Committee 
on Human Resources Development subsequently held hearings and issued 
reports to the House of Commons on the SIN. Both committees substantially 
agreed with our recommendations. 


1.23 HRDC created five working groups to address our recommendations 
for better management of the SIN. These groups were made up of 
representatives of the departments involved in the SIN program: HRDC, the 
CCRA, and Citizenship and Immigration Canada. The working groups 
reported to HRDC in February 2000. 


1.24 Our December 2000 follow-up reported that HRDC had taken several 
measures to improve the integrity of data in the Social Insurance Register; it 
had increased the number of SIN investigations and had other initiatives 
under way to address some of our 1998 recommendations. We indicated that 
a government-planned study on the impact and use of the SIN would provide 
essential information to help the government and parliamentarians make 
sound decisions about the future of the SIN, the advisability of restricting its 
use, matching of the data, and protection of privacy. 


1.25 Early in fall 2001, HRDC submitted a plan to the Treasury Board for 
improving the management of the SIN. The Department advised the Treasury 
Board that without significant new funding it would be unable to improve the 
integrity of data in the Social Insurance Register to an acceptable level, assess 
the effectiveness of current SIN policies, make its proof-of-identity 
requirements more rigorous, increase the number of SIN investigations, or 
undertake a national public awareness campaign. According to HRDC, the 
new funds it later received were not enough to carry out these activities. It 
did not reallocate its resources to fund them. 
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Role and importance of the SIN 


Focus of the audit 


1.26 The objective of our audit was to determine whether the government 
has safeguarded and strengthened the integrity of the Social Insurance 
Number. We examined its actions on the key concerns we raised in 1998 and 
the concerns of the Standing Committee on Public Accounts and the 
Standing Committee on Human Resources Development. More details 
appear in About the Audit at the end of the chapter. 


Observations and Recommendations 


Use of the SIN outside the federal government 


1.21 Responding in December 1999 to a recommendation by the Standing 
Committee on Human Resources Development, HRDC submitted a report 
confirming that the SIN’s primary role was as a file number in the 
administration of authorized federal programs. The government rejected the 
idea of legislation to restrict the use of the SIN outside the federal 
government. Recognizing concerns about privacy of personal information, 
including the SIN, in the private sector, it introduced the Personal Information 
Protection and Electronic Documents Act. 


1.28 That legislation provides Canadians with, among other things, the 
right to the protection of their personal information in the private sector. It 
defines the principles that govern the collection, use, and disclosure of 
personal information and provides a framework to address complaints of 
alleged breach of these principles. The Act (which will not be fully 
implemented until 2004) does not deal specifically with the SIN but captures 
it in the definition of personal information. It requires that an organization 
explain why it needs personal information and obtain the person’s consent to 
use it. This could serve to limit the use of the SIN. Prohibiting its use outside 
of the federal government, however, was not included in legislation. 


1.29 HRDC asked Statistics Canada to study the extent to which the SIN is 
used in the private sector but did not study its use in the voluntary sector or 
the public sector of the provinces and territories. The study by Statistics 
Canada concluded in February 2001 that SINs were being collected from 
employees and clients for a wide range of purposes. The study revealed the 
following: 


* 94 percent of employers ask for a SIN when hiring an employee; 


¢ 86 percent of employers do not know their obligations related to the 
SIN’s 900 series (issued to applicants who are not Canadians or 
permanent residents and who cannot work in Canada without an 
employment authorization from Citizenship and Immigration Canada); 


* 6 percent of all businesses surveyed and 27 percent of businesses with 
more than 250 employees have been given a false SIN by employees, 
although few reported any losses as a result; 
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* 10 percent of businesses surveyed have used SINs as employee file 
numbers; 


* 14 percent of all businesses surveyed and 57 percent of businesses with 
more than 250 employees have used SINs to identify their employees’ 
insurance and pension files; 


* 19 percent of businesses surveyed have shared SINs with insurance 
companies that provide coverage to employees (only 5 percent did so at 
the request of the employee); and 


* 9 percent of businesses surveyed have asked their clients for SINs and 
used them for a variety of purposes such as credit checks, proof of 
identity, and file numbers. 


1.30 Partly as a response to these survey results, HRDC undertook a number 
of public awareness activities on the SIN. However, their scope was limited by 
their primary focus on certain clients of the Department, particularly Canada 
Pension Plan beneficiaries and visitors to local HRDC offices. The 
Department also placed information on its Web site. It did not take other 
measures that would have reached a larger public, such as using the national 
media. HRDC has not measured the public awareness campaign’s 
effectiveness in informing individuals, employers, and other stakeholders 
about how and when to use the SIN. 


1.31 Recommendation. Human Resources Development Canada should 
assess the public awareness of how and when to use the Social Insurance 
Number appropriately and should carry out public education efforts as 
needed. 


HRDC’s response. As the Auditor General notes, HRDC did undertake a 
range of public awareness activities during 2001 and 2002. These included 
brochures, information to clients, and changes to the message on the SIN 
card itself. HRDC will assess the effectiveness of its Social Insurance Number 
public awareness activities during 2002—03. The Department will use the 
results of that assessment to guide the next phase of its public education 


efforts that will begin in 2003-04. 


Problems with the use of the SIN 


1.32 The Treasury Board policy implemented in 1989 reflects the 
government’s intention to restrict the use of the SIN by federal institutions to 
specified tax, pension, social, and benefits programs. The policy on the use of 
the SIN lists acts and regulations requiring its use and programs authorized to 
use the SIN. The Treasury Board Secretariat has not updated the policy 
governing the SIN since 1989. 


1.33. Although the SIN has been studied from many aspects since 1998, 
what has not been assessed is how federal programs use it and to what extent 
they rely on it. HRDC, for example, requires the SIN of applicants for benefits 
in its large programs. However, it is unclear whether HRDC uses the SIN as 
simply a number to manage a recipient’s file or also as a means to help 
establish recipient’s identity. 
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Integrity of the issuing process 


1.34 Although the government has reaffirmed its policy that the SIN is 
simply a file number and therefore not a form of identification, it has not 
given departments direction on acceptable ways to identify program 
beneficiaries, except for on-line services. Properly identifying clients is a 
fundamental control in any government program. 


1.35 The Treasury Board Secretariat became aware in 2000 that HRDC’s 
Opportunities Fund for Persons with Disabilities program was using the SIN 
without Treasury Board authorization. Follow-up work by the Department 
and the Secretariat found two other HRDC employment assistance programs 
(Youth Initiatives and Aboriginal programs) using the SIN without 
authorization. During our audit, the Department was attempting to remedy 
the situation by working closely with the Treasury Board Secretariat. It 
received the support of the Privacy Commissioner for using SINs in these 
programs. 


1.36 Further to its work in HRDC, the Secretariat decided to review 
whether federal institutions are respecting the policy that governs the use of 
the SIN. It will also consider revising the policy and developing guidelines for 
all federal institutions. The Secretariat expects to report the results of its 
review to the Treasury Board by June 2003. 


1.37 Recommendation. Human Resources Development Canada should 
use the SIN only where authorized by the Treasury Board. 


HRDC’s response. HRDC is working with the Treasury Board Secretariat 
and the Privacy Commissioner of Canada to include the three programs to 
which the Auditor General refers on the list of authorized uses. This will 
enable HRDC to fulfil its commitment to full compliance and underline the 
fact that the SIN is intended to serve solely as a file identifier for Government 
of Canada-related purposes. 


1.38. Recommendation. In its review, the Treasury Board Secretariat should 
look at why and how departments are using the SIN. 


Treasury Board Secretariat’s response. During its review, the Treasury 
Board Secretariat will closely examine the uses of the SIN by federal 
institutions and will make every effort to resolve any issues related to the 
Treasury Board policy concerning the SIN. 


Applying for a SIN 


1.39 =An application for a SIN is generally presented in person at one of the 
320 Human Resources Centres across Canada. An HRDC representative 
reviews the application for completeness and examines the document offered 
as proof of identity and status in Canada. The representative records relevant 
information from the document submitted and then stamps and initials the 
application. The stamp indicates to the Department’s National Services in 
Bathurst, New Brunswick (which issues all SIN cards) that 1 of the 38 
acceptable documents has been seen and the application is complete, 
accurate, legible, and valid. The application is then sent to National Services 
for review and for input in the Social Insurance Register, and the card is 
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issued. Applications can also be mailed directly to National Services with an 
original document or a certified photocopy. The process is illustrated in 


Exhibit 1.3. 


140 Over 1.2 million applications are processed each year; about | million 
new SINs are assigned and about 200,000 replacement cards issued. Since 
1998 the Department has issued over 3.7 million new and replacement SIN 
cards. 


Exhibit 1.3 Processing a SIN application 


SIN applications are made, for example, to request a SIN, a replacement card, or a change of name or status. 


New Brunswick 
vital statistics 


Citizenship and Immigration 
Canada 
(immigration records only) 


Application by mail Application in person 
or phone 


(New Brunswick only) 


National Services, Bathurst 


e Verifies HRCC certification of 
documents for in-person application 

e Reviews mailed application and 
supporting documents 

¢ Receives validation of information and 


Human Resources Centre of Canada 
(HRCC) 

¢ Reviews supporting documents 

e Verifies and completes application 

¢ Affixes certification stamp 

e Sends to National Services 


enters it in Social Insurance Register 
(SIR) 


Application approved 


i Investigation conducted , 
(if necessary) 


L aa = J 


Application rejected 


¢ SIN card mailed to applicant ¢ Additional information required 
e File amended in SIR 


SINs are issued without proper control 


141 We expected to find that HRDC was meeting the requirements of the 
Employment Insurance Act and Regulations in issuing SINs and that it had 
strengthened its controls over SINs. Instead, we found serious weaknesses in 
the issuing of SINs, and in our view HRDC is not doing enough to safeguard 
the SIN’s integrity. 


1.42. In our view, HRDC’s policies and practices do not meet, in important 
respects, the intent of the Employment Insurance Act and Regulations. Under 
the Act, the Canada Employment Insurance Commission is required to 
identify accurately all persons to whom it assigns a SIN. The Regulations to 


Report of the Auditor General of Canada—September 2002 


Chapter 1 | 11 


HUMAN RESOURCES DEVELOPMENT CANADA—THE INTEGRITY OF THE SOCIAL INSURANCE NUMBER 


the Act specifically require HRDC to determine both the identity and the 
citizenship status of SIN applicants. However, HRDC currently requires that 
a SIN applicant be asked for only one document as proof of both identity and 
citizenship. Many of the 38 different documents that HRDC accepts, such as 
birth certificates, cannot be linked to the person applying for a SIN. A 
document may show citizenship status, for example, but has no photograph or 
description of the owner to help identify the bearer of the document as its 
owner. In our opinion, determining identity properly would mean requiring 
additional identification. Consequently, our view is that HRDC’s policies and 
practices do not respect the intent of the Employment Insurance Act or 
Regulations. We are concerned that for the majority of the SINs issued since 
1998, the applicant’s identity and citizenship status were not checked 
properly because a single piece of identification is usually insufficient to check 


both. 


1.43 In 1998 we observed that many of the documents HRDC was 
accepting had significant drawbacks. We recommended that the Department 
assess the reliability of identity documents used to support applications for a 
SIN and that it take corrective action where needed. In our audit this year, 
we found that the Department had done little to strengthen its process of 
issuing SINs and was still reconsidering its approach to proof of identity and 
citizenship. It was accepting Quebec baptismal certificates or birth certificates 
issued before 1994, although the Government of Quebec had asked it to stop 
accepting these certificates because they are difficult to authenticate and 
could have been falsified. HRDC was also accepting as proof of identity and 
citizenship other documents with higher risk of falsification, such as expired 
Canadian passports no matter how old. However, at the end of our audit 
HRDC announced that it would stop accepting these documents as of 
August 2002, in order to strengthen the SIN proof- of-identity requirements. 


1.44 As already noted, HRDC does not require applicants to submit 
documents in the original; it accepts photocopies that have been certified as 
true copies by people in a wide range of occupations. The Department does 
not check the information, even on a sample basis, with the people who have 
certified the photocopies it accepts. In our view, photocopies represent a 
greater risk of falsification. Moreover, accepting photocopies circumvents one 
of the key controls in the processing of applications—the comparison by 
HRDC staff of documents presented by applicants with pictures of authentic 
documents and descriptions of their security features. 


1.45 For the most part, the Department does not have the means to contact 
the authority that issued the documents to check their validity. Exceptions 
are the checking of immigration documents with Citizenship and 
Immigration Canada and of birth certificates with the Vital Statistics Branch 
of New Brunswick’s Department of Health and Wellness—which together 
account for about 30 percent of applications that HRDC processes. These 
checks verify that the document is valid but do not confirm that the bearer of 
the document is its rightful owner. 
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146 We also found that the Department’s policies on issuing SINs focus less 
on safeguarding the integrity of the application process and more on satisfying 
the applicant. For example, HRDC staff are advised to use discretion when 
they attempt to confirm an applicant’s identity if presented with documents 
of questionable validity, in case the applicant views their attempts as 
harassment. Moreover, HRDC’s rules are unclear in many respects and are 
therefore subject to misinterpretation. We found examples of contradictory 
statements and ambiguous instructions. 


147 During our audit, HRDC completed a study to identify ways to 
improve how it issues SINs while reducing the processing time. The 
Department informed us that it intends to use the results of this study in its 
planned changes to its process. 


Staff lack adequate training, tools, and information 


1.48 All HRDC staff should be processing SIN applications in a consistent 
way. However, we found that job responsibilities, training, tools, and 
information for issuing SINs vary greatly among local HRDC offices. There is 
a risk that how much attention a SIN application receives depends on where 
it is submitted and who processes it. 


1.49. Responsibilities may not be understood clearly. For example, while all 
staff are responsible for fraud detection, most front-line staff we interviewed 
said they tend to rely on National Services in Bathurst to detect fraud. For 
applications handled initially by HRDC’s local offices, however, National 
Services relies on the local staff to detect fraud. And despite high rate of 
turnover in some offices, no formal training is given to staff before they start 
handling applications. Computer-based training is available on the 
Department’s internal network. However, we observed in the offices we 
visited that new employees were often left to learn on the job. 


1.50 HRDC has made some progress in providing staff with tools for fraud 
detection. For example, it introduced risk indicators for staff of local offices to 
use on incoming applications. It also issued a guide containing pictures and 
descriptions of security features of authentic documents for comparison with 
documents presented by applicants. In early 2002, it supplied regional offices 
with ultraviolet lamps (to help detect fraudulent documents) for distribution 
to all local offices. When we visited local offices, however, we noted that 
some had not obtained the lamps and that generally staff had received no 
training in their use. 


Inadequate control over SINs issued to applicants who are not Canadians or permanent 
residents 


1.51 HRDC also issues SINs to short-term visitors, refugee claimants, 
seasonal workers, and foreign students. A SIN can also be issued to citizens of 
other countries, even if they are not visiting or residing in Canada. These 
numbers begin with a 9 to distinguish them from regular SINs and to indicate 
that the bearer requires separate authorization to work in Canada. When a 
person who has been issued a 900-series SIN is granted Canadian citizenship 
or becomes a permanent resident and applies for a regular SIN, HRDC 
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cancels the 900-series SIN and assigns a regular number in its place. Over 
time, HRDC has issued close to 1.6 million 900-series SINs. At the end of 
2001, the Social Insurance Register showed more than 900,000 of these SINs 


that had not yet been cancelled. 


1.52 Since our 1998 audit, the Department has made no changes in the way 
it controls the 900-series SINs. We found that, as in the case of regular SINs, 
HRDC does not take adequate steps to establish the identity of people who 
apply for 900-series SINs. It requires only one document to prove both 
identity and citizenship. It will accept any one of a range of documents 
produced by the Government of Canada (such as immigration forms) and by 
foreign governments (such as passports). While passports demonstrate both 
identity and citizenship, many other documents do not. And staff have no 
training or tools to verify the validity of documents issued by foreign 
governments. Moreover, HRDC will accept photocopies of these documents 
(including foreign passports). Detecting fraud is more difficult when working 
with photocopies. 


1.53 Furthermore, under the Regulations to the Employment Insurance Act, 
HRDC is supposed to require applicants for 900-series SINs to show why they 
need one. Some of the identity and citizenship documents HRDC accepts 
(for example, an authorization to work in Canada) might suggest why the 
person would need a SIN, but most indicate only that the person is allowed to 
be in Canada. We found that the Department does not require staff to get an 
explanation from all 900-series applicants as to why they need a SIN (see the 
case study below). Since 1998 the number of 900-series SINs assigned each 
year has been increasing. According to HRDC, this trend is consistent with 
the changes that Citizenship and Immigration Canada has reported in the 
temporary resident and refugee claimant population. In 2001, over 110,000 of 
the 900-series SINs were issued; this was 41 percent more than in 1998. 


Processing 900-series SIN applications 


This case illustrates the frustrations that some staff face at Human Resources Centres 
of Canada in processing 900-series SIN applications. 


A front-line worker in a large Canadian city began to question why individuals who 
were in Canada as visitors for short periods of time were requesting and receiving SIN 
cards. Despite expressing concern that SINs were being issued to individuals who did 
not need them, he was instructed by management to process the applications. He did, 
but submitted them to National Services without including the required certification 

stamp or the numbers of the submitted documents and provided for each application a 
photocopy of the passport and the entry stamp, hoping that this would cause National 
Services to reject the applications. National Services did not reject the applications; it 
approved them and issued the SIN cards to visitors to Canada. 


1.54 Asin 1998, the 900-series numbers and the cards do not expire. Most 
people holding them are expected either to be in the country only temporarily 
or to gain permanent status in Canada and be granted a regular number. We 
noted during our audit, however, that more than 60 percent of 900-series 
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SINs that have not been cancelled were issued more than five years ago. It is 
likely that the original holders have left the country, but there is no way of 
knowing what happened to their cards. 


1.55 Recommendation. Human Resources Development Canada should 
ensure that its policies and practices respect the intent of the Employment 
Insurance Act and Regulations, which require it to 


+ determine the identity and citizenship status of applicants for a Social 
Insurance Number; and 


* obtain proof of the need for a Social Insurance Number from applicants 
who are not Canadian citizens or permanent residents. 


1.56 Recommendation. Human Resources Development Canada should 
reconsider its goals for the 900-series SIN and the associated risks. It should 
revise its policy and practices accordingly. 


1.57. Recommendation. Human Resources Development Canada should 
improve the integrity of its SIN application process. In particular, it should 


* assess the reliability of the identity and citizenship documents it accepts 
as support for applications and implement corrective action where 
necessary; 


* make reasonable efforts to develop a means of checking the validity of 
identity and citizenship documents with the authorities that issued 
them; and 


* ensure that staff have the training and tools to comply with the 
Employment Insurance Act and Regulations when they process 
applications for a Social Insurance Number. 


HRDC’s response. HRDC is building on previous initiatives to improve the 
integrity of the issuing process to ensure that it is meeting the Social 
Insurance Number goals of the Employment Insurance Act. 


As of August 1, 2002, the Department implemented a uniform standard for 
Canadian-born applicants when it stopped accepting baptismal certificates 
from applicants born in Quebec and Newfoundland and Labrador. HRDC 
will build on this step by implementing its decision to render certified 
photocopies ineligible as proof of identity by November 1, 2002. This means 
that HRDC will accept only original documents to prove identity for 
applications made in-person or through the mail. The Department will 
accelerate the necessary training of staff on policies and procedures, including 
the new proof-of-identity requirements. Initiatives such as the use of 
ultraviolet lights to examine original documents, and improved staff training, 
enhance HRDC’s security measures. 


In February 2002, the Department reviewed how SIN applications are 

processed and is considering ways to further strengthen proof of identity, such 
as implementing a guarantor system similar to the one used for the Canadian 
passport; or including photo identification as part of the application process. 
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Integrity of the Social Insurance 


Register 


HRDC will continue with efforts that it has been making since 1998 to access 
vital statistics databases of the provinces and territories. It should be 
recognized that, with privacy and security concerns, these partnerships take 
time. Eventually, as is the case right now in New Brunswick, this will allow 
the Department to directly, accurately, and consistently verify birth 
information provided by Canadian-born SIN applicants with the province or 
territory of birth. HRDC is currently working toward a pilot project with 
British Columbia to verify the information provided on a SIN application 
with the provincial vital statistics agency. 


HRDC is taking steps to amend the Employment Insurance Regulations to 
improve the management of SINs needed by people who are neither 
Canadian citizens nor permanent residents (900-series). Under this change, 
these 900-series SINs will be limited to an amount of time that is linked to 
the individual’s authorized stay in Canada, rather than indefinitely. These 
SIN cards will display the expiry date of that Social Insurance Number. 


HRDC also recognizes the need to address the issue of existing 900- series 
SINs. In order to bring about consistency, and after regulatory approval is 
obtained, existing 900-series SIN cardholders will have a reasonable period of 
time to provide acceptable proof of identification and proof of need to obtain 


a new card. 


The Department anticipates that these measures represent additional 
significant improvements in the overall security and accuracy of the Social 
Insurance Number database. 


1.58 The Social Insurance Register is the database that records the personal 
information provided by individuals who apply for a SIN. It contains all the 
SINs issued so far and basic personal information such as name, date of birth, 
father’s name, and mother’s maiden name. In 1998 we reported that the 
integrity of the Register was a major concern. We expected to find in this 
audit that HRDC had largely resolved this concern and ensured that the 
information in the Register was reasonably accurate and complete. 


Action taken to date 


1.59 In 1998 we recommended that HRDC set out a plan for bringing the 
reliability and completeness of the Register up to an appropriate level. In 
2002 we found that the Department has not carried out a comprehensive 
approach for improving the integrity of the information. According to 
HRDC, a lack of funding prevented it from implementing the improvement 
plan it had originally developed in response to our 1998 concerns. At the end 
of our audit, there was still no standard for the quality of information in the 
Register. 


1.60 Nevertheless, HRDC has attempted to improve the Register through 
various initiatives. The Department has compared data in the Register with 
the Old Age Security and Canada Pension Plan (CPP) records of almost 

10 million people, and personal information has been updated accordingly. 
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This included 


* cross-checking CPP and Old Age Security files against at least 3 million 
SIN records for which HRDC had never obtained proof of identity; 


* noting on 101,000 SIN records that the individual had died; and 
* changing about 276,000 incorrect dates of birth. 


1.61 The Department undertook various initiatives to obtain more accurate 
information on deaths. Since 1998, it has received over 4.8 million death 
notices from different sources, such as the Canada Customs and Revenue 
Agency and the Old Age Security and Canada Pension Plan programs. 


1.62. The Department also implemented SIN Tele-App which allows clients 
in New Brunswick to apply for a SIN by telephone. Their information is 
validated against the database at the Vital Statistics Branch of New 
Brunswick’s Department of Health and Wellness or against Citizenship and 
Immigration Canada’s immigration database. 


163 =HRDC has built into its system some indicators to detect potentially 
fraudulent applications. However, it could add other indicators of unusual 
activity, such as SIN cards mailed to a post office box, cards returned to 
National Services and never claimed, or multiple cards sent to the same 
address. A special study by the Department in 2001 found that in many cases 
HRDC had mailed more than 100 SIN cards (with different numbers) to a 
single address over a period of a year (including 225 cards to one address). 
Because the Department does not track such activities, it did not detect these 
events as it processed the applications; nor did it investigate them afterward. 


Information in the Register can be improved further 


1.64. One of the main difficulties in maintaining the Register is that HRDC 
is not advised routinely of births or changes of name. Although HRDC 
receives death notices from many sources, it indicates that this information is 
not comprehensive. The integrity of the Register would be improved if the 
Department had access to provincial and territorial vital statistics. During our 
audit, HRDC initiated discussions on this subject with the Vital Statistics 
Council, an interjurisdictional advisory group comprising the heads of the 
vital statistics divisions/agencies from all the provincial and territorial 
governments and the Health Statistics Division of Statistics Canada. The 
Council provides a forum for developing common approaches to collecting 
vital statistics, sharing information with outside parties, and facilitating 
problem solving for issues related to vital statistics. 


1.65 However, HRDC has not pursued other sources of information that 
could improve the Register. For example, it lacks formal agreements with 
federal departments that issue identity and citizenship documents, such as 
citizenship cards and passports. While HRDC compares monthly Register 
data with Old Age Security data, it has not yet done so with the Canada 
Pension Plan or the Quebec pension plan (Régime de rentes du Québec). At 
the end of our audit, HRDC was involved in discussions with the CPP and 


Report of the Auditor General of Canada—sSeptember 2002 Chapter 1 17 


HUMAN RESOURCES DEVELOPMENT CANADA—THE INTEGRITY OF THE SOCIAL INSURANCE NUMBER 


18 


| Chapter 1 


the Quebec pension plan. As already noted, HRDC has made efforts to 
improve the Register’s information on deceased people. However, many of 
the death notifications it receives are not checked with provincial or 
territorial vital statistics. 


Dormant file review 


1.66 HRDC classified over 3.2 million SINs as dormant after obtaining 
evidence from the Canada Customs and Revenue Agency that the SINs 
belonged to people over the age of 23 who had not filed a tax return, or for 
whom no income had been reported from 1993 to 1997. 


1.67 The intent of this exercise was to help explain the gap between the 
number of SINs and the population and to flag SINs for greater scrutiny if the 
Department were to receive an application for a replacement card or any 
other changes. Until May 2002, however, National Services was processing 
transactions involving dormant SINs without investigating them. It simply 
removed the dormant marker from the file and took no special action to 
ensure that the transaction was legitimate. 


1.68 In May 2002, HRDC began routinely investigating SIN-related 
transactions in dormant files. These dormant SINs can still be used as a first 
step to access benefits such as Employment Insurance, Canada Pension Plan, 
and Old Age Security—because federal programs that use SINs are not 
informed of those that are dormant. HRDC advised us that other information 
in addition to the SIN is required to establish eligibility for its benefits or 
entitlement to them. 


1.69 The Department has found problems in the way the dormant file 
exercise was carried out. For example, the results excluded SINs that should 
have been identified as dormant: they did meet the screening criteria. In our 
view, the identification of dormant SINs has done little so far to reduce the 
gap between the number of usable SINs and the size of the population. By a 
usable SIN, we mean one that can be used as an account number in 
transactions with the federal government and not raise any concern or 
suspicion in itself. 


Problems with information integrity persist 


1.70 There are still significant discrepancies between the number of usable 
SINs and Statistics Canada’s estimates of the Canadian population over 

20 years old. While the Social Insurance Register is not a census of the 
Canadian population and comparing it with Statistics Canada’s census 
presents some problems, the question remains: Why would there be millions 
more usable SINs than Canadians who could use them? 


1.71 These discrepancies concern us, as they did in 1998. Despite HRDC’s 
efforts to improve its information on SIN holders, the Register still contains 
significant variances from population estimates—variances that have 
increased over the past four years (Exhibit 1.4). While excluding the dormant 
SINs would reduce these discrepancies from 5 million to 2.4 million, we 
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included them in the exhibit because they are still usable as a first step to 
access federal benefits. In HRDC’s view, dormant SINs are unlikely to be used 
in the future. 


1.72 However, even with the dormant SINs removed, 2.4 million is a 
significant gap. The Department suggests that these excess SINs probably 
represent people who have left the country (most often, holders of 900-series 
SINs), deaths of which HRDC has not been notified, people with multiple 
SINs, and frauds; or they could reflect the timing of the census. In our view, 
HRDC needs to explain the variances fully. 


Exhibit 1.4 Comparing the Social Insurance Register with the population (2001 Census) 


Usable SINs in the Population based on 
Social Insurance Statistics Canada’s 
Age : Register | 2001 Census Variances 

| 20-29 4,378,563 3,854,010 524,553 
30-39 5,390,632 4,619,595 TI MOST 
40-49 5,838,149 4,912,295 925,854 
50-59 4,656,062 3,10 SSIS) 976,067 
60-69 2,899,245 2,407,405 491,840 
70-79 2a53;021 1,822,880 SSOMAl 
80-89 1,102,428 ED ZS 304,503 
90 years or more 829,687 134,120 | 695,567 
Total 27,247,787 22,228,225 5,019,562 


Source: Human Resources Development Canada and Statistics Canada 


1.73 Another issue identified in this audit is the number of SINs that have 
never been cross-checked with other departments or supported by any proof- 
of-identity documents. These SINs were issued before the introduction in 
1976 of the proof-of-identity program. The SIR shows over 8.3 million of 
these usable SINs. This represents a significant decrease from 1998. HRDC 
has accomplished this mainly by obtaining information from the CPP and Old 
Age Security programs. Due to the cost and inconvenience to SIN holders, 
HRDC does not plan to certify the remaining SINs systematically but only 
when there is a new transaction. However, it hopes it will be able to certify 
many more of these SINs if and when it is given access to provincial and 
territorial vital statistics or other official sources of information. 


HRDC needs to better report on its improvements to the Register 


1.74 Weare concerned that HRDC reported the improvements it has made 
to the Social Insurance Register in a way that could be misinterpreted. 
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1.75  Inits March 2000 Performance Report, HRDC said it had closed the gap 
between the number of SINs and the population to 800,000. It arrived at that 
figure by assuming that all holders of dormant SINs had died or left the 
country, although it had no proof that either was the case. Moreover, the 
identification of dormant SINs simply provided a possible explanation for a 
large portion of the gap between data in the Register and population 
estimates at that time. HRDC had taken no steps to actually reduce or close 
the gap. The dormant SINs could still be used as a first step to access benefits. 


1.76 In its December 1999 response to the Standing Committee on Human 
Resources Development, HRDC implied that by identifying dormant SINs it 
had reduced the number of uncertified SIN records by 2.6 million. In our 
view, there is no relationship between dormant SINs and uncertified SINs. 
Dormant SINs are those that have not been used for some time, whereas 
uncertified SINs are those for which the personal information has not been 
verified. Declaring a SIN to be dormant does not somehow validate the 
personal information that HRDC has on record. 


1.77 Recommendation. Human Resources Development Canada should 
set goals for the completeness and reliability of the Social Insurance Register 
and take the steps necessary to meet them. 


HRDC’s response. HRDC has made numerous improvements to its Social 
Insurance Number database, the Social Insurance Register (SIR), to make it 
more complete and reliable. Since 1998, HRDC has certified the accuracy of 
millions of SIN records. The Department has also identified 3.2 million other 
SINs that have not been used in at least five years, which includes 600,000 
SINs of deceased individuals. HRDC conducts investigations as appropriate if 
someone then uses those Social Insurance Numbers to apply for Employment 
Insurance or is seeking a replacement SIN card. 


To provide greater security, HRDC is taking steps to amend the Employment 
Insurance Regulations to cancel SINs that have not been used in five years. 
Once a SIN is cancelled, a person will be required to provide acceptable proof 
of identity to reactivate that SIN. 


The Department has initiated a comprehensive review of the Social 
Insurance Register. That review will assist HRDC to set goals for SIR 
integrity, define the actions and timeframes needed to meet those goals, and 
implement the additional required changes. This process will be completed 


by April 2003. 


1.78 Recommendation. Human Resources Development Canada should 
ensure that its reporting of improvements to the quality of the information in 
the Social Insurance Register is clear and unambiguous. 


HRDC’s response. HRDC agrees that reporting should be clear and 
accurate. It has already started to review this reporting to ensure that it 
provides information as clearly as possible. 
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Investigating fraud  Arisk-based approach to investigation and control is needed 


179 We expected that HRDC would link its fraud investigations to the risks 
associated with the issuing of the SIN and with its use by the federal 
government. 


180 Instead, we found that HRDC has completed no risk-based analysis of 
the SIN program. Nor has it attempted to determine the extent and nature of 
SIN-related fraud. Its level of effort is based mainly on the number of SINs 
issued, the resources it has available, and a number of specific risk indicators. 
Although the number of SIN investigations has increased since 1998 
(Exhibit 1.5), it is not clear that this effort was focussed on high-risk areas. 
Furthermore, since current levels of effort reflect the supplementary funding 
the Department received, SIN investigations may not remain at this level 


after this funding ends in 2002-03. 


Exhibit 1.5 Completed investigations of suspected SIN fraud 


Investigations 
8,000 


? 1998-99 11999-2000 2000-01 2001-02 


Source: Human Resources Development Canada 


181 In 1998, we recommended that HRDC redesign its performance 
indicators for SIN investigations to recognize their importance not only to the 
Employment Insurance program but also to other federal programs and 
provincial, territorial, and municipal programs. In 2001-02, HRDC began the 
setting of goals and the routine tracking of the number of SIN investigations 
carried out by its local offices. This performance indicator does not capture 
information on the quality and results of those investigations. However, since 
1998 the SIN Investigation Service has reviewed most SIN investigations for 
completeness. 


1.82 Recommendation. Human Resources Development Canada should 
adopt a more comprehensive risk-based approach to investigating SIN-related 


fraud. 


HRDC’s response. The Department has implemented the SIN Application 
Review Program (SARP), a risk-based program that guides investigation 
priorities related to SIN applications, which forms part of HRDC’s approach 
to risk management. The Department will determine if SARP can be 
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expanded to guide priority-setting for other SIN investigations. Any risk- 
based approach will recognize that the Department needs to work closely 
with federal and provincial government officials, police services, employers, 
and other partners in combating SIN-related fraud and abuse. 


Investigators lack structured training and adequate tools 


1.83 In this audit, we expected to find that SIN investigators were receiving 
more training and support than in 1998. However, we found that there is still 
no formal training program. While they now have access to some computer- 
based training, investigators informed us that they would benefit from 
classroom instruction. 


1.84 We also expected that SIN investigators would have access to adequate 
tools. We found that various tools are being made available: on-line access to 
certain databases, a guide containing pictures and descriptions of security 
features of authentic documents, and ultraviolet lamps to help identify 
fraudulent documents. However, as we found with front-line staff, 
investigators in many of the local offices we visited had not yet received the 
lamps and some had not received training in the use of the tools. 


1.85 Recommendation. Human Resources Development Canada should 
ensure that investigators have the proper training and tools. 


HRDC’s response. HRDC agrees with the recommendation. The 
Department has developed an integrated course on Social Insurance Number 
investigation and identity fraud. The first session of the course will take place 
in the fall of 2002. All HRDC offices have now received ultraviolet lamps and 
the necessary instructions and training to help identify fraudulent documents 
through their use. This is matched by a regularly-updated guide to identity 
documents. The decision that HRDC will no longer accept certified 
photocopies to prove identity from people applying for SINs in-person or 
through the mail will increase the effectiveness of these tools. 


Conclusion 


1.86 The SIN is used widely outside the federal government as well as 
inside. It is one of a handful of personal identifiers that a person needs to 
function in Canadian society. It is used in several major federal benefits 
programs and in transactions with the Canada Customs and Revenue 
Agency. It is used often, but not always, by provinces, cities, and the financial 
services industry. It also appears that many businesses use it when offering 
credit to their clients. All of these uses give the SIN its value. 


187 We began our audit expecting to find that the issues we raised in 1998 
had been substantially addressed. Instead, we have concluded that HRDC 
has not done enough to safeguard and strengthen the integrity of the SIN. 


1.88 Ensuring that SINs are issued to only those eligible is still problematic. 
Most of the application-processing weaknesses we found in 1998 have not 
been addressed. In our view, HRDC’s policies and practices do not meet, in 
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important respects, the intent of the Employment Insurance Act and 
Regulations. We are concerned that for the majority of the SINs issued since 
1998, the applicant’s identity and citizenship status were not checked 
properly because a single piece of identification is usually insufficient to check 


both. 


189 The integrity of the Social Insurance Register also remains a persistent 
problem. While HRDC made a number of improvements after 1998, there are 
still several weaknesses that it has not dealt with adequately. The number of 
usable SINs for people over 20 years old exceeds the census figure by 

5 million. Although HRDC has designated millions of SINs as dormant, they 
could still be used to access federal benefits programs without triggering an 
investigation. Finally, there are still 8.3 million records in the Social 
Insurance Register that have never been cross-checked with other 
departments or supported by any proof-of-identity program. These uncertified 
SINs are also usable. 


190 Since 1998 the Department has stepped up the number of SIN-related 
fraud investigations. However, there is little basis for judging whether the new 
level of effort is adequate—because there is little information on the extent of 
SIN-related fraud. The Department has not done a comprehensive risk 
analysis as a basis for its investigation efforts. 


HRDC’s comments. HRDC is committed to fulfilling its role as part of the 
overall Government of Canada response to Canadians’ heightened awareness 
of security issues. The Department recognizes the priority of addressing the 
security issues related to personal identification documents and has 
responded with actions that will strengthen its ongoing efforts to 


* ensure appropriate issuance of SINs; 


+ improve the integrity of the key SIN database, the Social Insurance 
Register; 
* vigorously pursue SIN fraud and abuse; and 


* raise awareness that the Social Insurance Number (SIN) is authorized 
only as a file identifier for specific Government of Canada-related 
purposes and how to protect against SIN misuse. 


HRDC is dedicated to diligent stewardship of the Social Insurance Number. 
Since 1998 the Department has taken steps to improve the management of 
the SIN and the integrity of the Social Insurance Register (SIR). The 
Department agrees with the recommendations in this chapter, and as 
indicated in its responses, is taking immediate action to implement a number 
of key initiatives. These actions will improve the policies, processes, and 
practices of its management and administration of the Social Insurance 
Number. We will continue to work with the Office of the Auditor General 
and other partners on further initiatives that fully meet the recommendations 
of the report and the expectations of Canadians. 
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About the Audit 


Objective 


The objective of our audit was to determine whether the federal government was safeguarding and strengthening the 
integrity of the Social Insurance Number (SIN). We assessed the extent of the government's action to address the 
key concerns we raised in our 1998 audit and those raised by the Standing Committee on Public Accounts and the 
Standing Committee on Human Resources Development. 


Scope 


Using the problems we identified in 1998 as a baseline, we tested whether they still existed and were still significant 
in 2002. We looked at four issues: 


¢ the role and importance of the SIN; 
¢ the integrity of the information in the Social Insurance Register (SIR); 
* control over the issuing of SINs and the SIN cards; and 


¢ efforts to combat fraud. 


We carried out our audit mainly at HRDC headquarters in the National Capital Region; at Bathurst and Moncton, 
New Brunswick; in regional and local HRDC offices; and at the Treasury Board Secretariat. We gathered additional 
information from the Canada Customs and Revenue Agency, Citizenship and Immigration Canada, the Office of the 
Privacy Commissioner, Statistics Canada and the Royal Canadian Mounted Police. 


Criteria 


Legislative and other authorities should provide a sound framework for safeguarding and strengthening the integrity 


of the SIN. 


Human Resources Development Canada should be aware of how the SIN is used in the private sector and the public 
sector in provinces and should lead action to encourage its proper use. 


The Treasury Board Secretariat should monitor departmental compliance with its policies on the use of SIN and data 
matching. 


HRDC should make reasonable efforts to ensure that it provides new SINs and replacement cards only to eligible 
individuals. 


HRDC should make reasonable efforts to ensure that the Social Insurance Register is accurate, complete, and secure. 


Fraud detection and investigation efforts should be based on risk. 
Ratings 
We assessed the action of departments/agencies against our original audit recommendations (see Key Message at the 
beginning of the Chapter). We used the following ratings: 
* Completed. Corrective action has been fully implemented 
* Satisfactory progress. Progress is being made at a satisfactory pace 
* Limited progress. Some progress is being made, but the pace or scope is not satisfactory 


No progress. No evidence of progress although the department or agency accepted the recommendation from 
the original audit 


Rejected. The department or agency did not accept the recommendation from the original audit 


* Unknown. Status of progress is unknown or information is not available 
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Audit team 


Assistant Auditor General: Maria Barrados 
Principal: Peter Simeoni 
Director: Suzanne Therrien 


Christian Couture 
Guy Dumas 
Susan Gomez 

Lise Tremblay 


For information, please contact Communications at (613) 995-3708 or 1-888-761-5953 (toll-free). 
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Key Message 


21 Health Canada has made limited progress toward resolving some of the weaknesses identified in our 1999 
audit. However, national surveillance is still weak; many systems still lack timely, accurate, and complete disease 
information; and gaps in surveillance continue. These weaknesses, taken together, compromise Health Canada’s 
ability to anticipate, prevent, identify, respond to, monitor, and control diseases and injuries. Further, they 
compromise its ability to design, deliver, and evaluate public health activities. 


ORIGINAL ISSUES PROGRESS RATING* 


2.2 Health Canada should provide strong leadership Health Canada is taking a greater leadership role in 
and address factors that are compromising its ability to national health surveillance: 


engage in national health surveillance: 
eg e It has made progress in establishing a national 


¢ Lack of federal public health legislation or a framework for health surveillance. This framework 
national framework to link the separate public allows for greater collaboration among the partners LIMITED 
health activities in the provinces and territories. responsible for health surveillance. We are PROGRESS 


concerned that there are still only a few 
agreements on data sharing between Health 
Canada and the provinces and territories, and no 
agreement on common standards and nationally 
reportable diseases. 


No agreement among the various partners on 
clearly defined roles and responsibilities in the 
areas of data collection and dissemination and 
responding to public health threats. 


2.3 Health Canada, in collaboration with the provinces | Health Canada’s progress in this area has been 
and territories, should improve the timeliness, accuracy, inconsistent: 
and completeness of disease information. 


LIMITED 


¢ Many of its surveillance systems still lack timely, 
PROGRESS 


accurate, and complete disease information. 
Exceptions include surveillance systems for enteric 
diseases, influenza, and AIDS. 


Progress has been made on The Canadian 
Integrated Public Health Surveillance Project, 


reg ees SATISFACTORY 
which is intended to address these concerns. PROGRESS 


However, it still faces some challenges and will not 
likely be fully operational for some time. 


2.4 Health Canada, in collaboration with other Health Canada has made significant progress in 

organizations, should take steps to fill gaps in developing a national diabetes surveillance system. LIMITED 
surveillance of chronic disease, such as diabetes and However, the gaps in other chronic disease surveillance PROGRESS 
cardiovascular disease. continue. 


*Possible ratings are completed, satisfactory progress, limited progress, no progress, rejected, unknown. (See About the Follow-Up for an explanation of the 
ratings.) 
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2.5 For the most part, diabetes and breast cancer are the only chronic diseases with a national surveillance system. Despite the 
recognized importance of surveillance for chronic diseases, limited resources have been allocated to this area and health 
surveillance information is lacking for cardiovascular disease, musculoskeletal disease, cancer (except breast cancer), chronic 
respiratory disease, mental illness, and injuries. Further, surveillance is also lacking for risk determinants and for impacts of 
interventions, screening, and treatment on health outcomes. 


2.6 Health Canada should identify its health surveillance priorities and ensure that adequate and stable funding is available to 
develop and maintain the surveillance systems that it identifies as priorities. 


Health Canada has responded. Health Canada has agreed to continue working on the recommendations and 
taking corrective actions as described in its responses, included in the chapter. 
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Introduction 


Health surveillance: Essential for public health activities 


27 Health surveillance is a core function of public health activities. It 
begins with the ongoing collection of information on disease, which is then 
integrated, analyzed, and interpreted. The data these activities produce are 
disseminated in a variety of formats to those who need the information so 
they can take appropriate action. 


28 Information informs decision makers and is crucial to their making the 
right decisions at the right times. Information obtained through health 
surveillance is important for understanding the health status of a population. 
Authorities can use health surveillance information to identify a disease 
outbreak and respond to it and to prevent disease by educating the public 
about factors that may put health at risk (that is, risk determinants, such as 
lack of exercise). Further, the information can be used to monitor the 
incidence of disease and to help control disease by determining the most 
effective treatment. Health surveillance information is also needed to develop 
government health policies and programs, to measure the performance of 
government activities and evaluate their effectiveness, and finally, to assess 
the quality and accessibility of health care. 


29 Provincial, territorial, and municipal governments as well as public 
health professionals and laboratories are closely involved in health 
surveillance. As primary providers of public health services, they supply 
Health Canada with disease information for national health surveillance. 
Health Canada’s role is to provide leadership and support for health 
surveillance across the country. It is responsible for gaining the collaboration 
of the various partners in health surveillance, integrating the disease 
information it receives from them, developing the surveillance systems to 
analyze and interpret the information, and then disseminating the results. 
Other partners in health surveillance include Statistics Canada, the 
Canadian Institute for Health Information, the Canadian Food Inspection 
Agency, and several organizations outside government. 


210 Collaboration is critical to national health surveillance, given the many 
partners it involves. Working together allows for timely information sharing, 
co-ordinated responses, and effective action. 


2.11 The risks that poor health surveillance creates are very real: 
preventable illnesses may not be prevented, approaches to treating disease 
may not be as effective as they could be, and government funding may be 
directed at the wrong health issues. For example, in 1999 we reported on a 
salmonella outbreak during which the exchange of information was not as 
timely and collaboration not as effective as they could have been, while the 


illness spread. 
212 Health Canada’s Population and Public Health Branch is responsible 


for national health surveillance. The Branch was created in a reorganization 
of Health Canada in 2000. In addition to health surveillance and disease 
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control, the Branch is responsible for policies, programs, and systems for 
disease and injury prevention, health promotion, community action, and the 
Department’s emergency preparedness and response capacity. Its 
responsibilities include most elements of the former Laboratory Centre for 
Disease Control and the former Health Promotion and Programs Branch, as 
well as some responsibilities of the former Health Protection Branch. Our 
Office reported on the Branch’s population health activities (disease and 
injury prevention, health promotion, and community action) in December 


2001. 


213 The Branch is divided into several centres. Those mainly responsible 
for surveillance include the Centre for Surveillance Coordination, the Centre 
for Infectious Disease Prevention and Control, the Centre for Chronic 
Disease Prevention and Control, the Centre for Healthy Human 
Development, the Centre for Emergency Preparedness and Response, the 
National Microbiology Laboratory, and the Laboratory for Food-borne 
Zoonoses (diseases that animals can transmit to humans through food). 


214 In 2001-02, the Population and Public Health Branch had a budget of 
$365 million. That included $66 million for salaries and $13 million for 
benefits of 1,146 full-time staff equivalents, and $78 million for operating 
expenses. It also included $208 million for grants and contributions, most of it 
destined for population health activities rather than health surveillance 
activities. 


Concerns we raised in 1999 


2.15 In 1999 we examined the way Health Canada carried out national 
health surveillance activities and how those activities supported the other 
components of public health—health assessment, health promotion, disease 
and injury prevention, and health protection. 


2.16 Our report raised a number of concerns about the Department’s 
national health surveillance activities. We concluded that our concerns were 
significant and called for corrective action in several areas. 


2.17 Specifically, we were concerned that there was no mechanism for 
assigning roles and responsibilities between Health Canada and its provincial 
and territorial partners. We also identified weaknesses in both the disease 
information forwarded to Health Canada and the Department's 
dissemination of surveillance information in its various forms. We noted gaps 
in surveillance activities in areas where surveillance systems were inadequate. 
Finally, we were concerned about the way Health Canada had evaluated, 
measured, and reported the performance of its national health surveillance 
activities. 


Focus of the follow-up 


2.18 The objective of this follow-up was to determine the progress made in 
addressing the observations and recommendations contained in our 1999 
Report—Chapter 14, National Health Surveillance: Diseases and Injuries; 
and Chapter 15, Management of a Food-borne Disease Outbreak. We also 
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followed up on recommendations made by the Standing Committee on Public 
Accounts. We looked at the Department's actions and the commitments it 
had made in response to our audit; and we reviewed the progress it had 
reported to the Public Accounts Committee. In the last three years, Health 
Canada has undertaken a number of new initiatives related to national health 
surveillance, which we also reviewed. Our report presents the results of the 
areas that were re-audited. Further details about the follow-up’s objective, 
scope, and criteria can be found in About the Follow-Up at the end of the 
chapter. 


Observations and Recommendations 


Improving collaboration on national © 219 In 1999 we noted that a survey of public health professionals at the 
health surveillance provincial, territorial, and regional levels revealed that participants wanted 
Health Canada to take a more proactive role in health surveillance. They 
agreed that strong leadership was needed from Health Canada. 


2.20 Atthat time, we also noted two factors that were compromising Health 
Canada’s ability to conduct national health surveillance: 


¢ There was neither federal public health legislation nor a national 
framework to link the separate public health activities of the provinces 
and territories. 


¢ There was no agreement among the various partners on clearly defined 
roles and responsibilities for collecting and disseminating data and for 
responding to public health threats. 


Progress in developing a national framework 


2.21. Each province and territory has its own public health legislation and 
public health priorities, resulting in 13 separate public health systems. Making 
progress in this complex environment can be a challenge. Therefore, strong 
national leadership is needed. 


2.22 In 1996, Health Canada recognized that because each province and 
territory has its own public health legislation, a national framework would be 
needed if national health surveillance were to succeed. In its response to our 
1999 audit report, Health Canada noted that it was reviewing its legislation 
with a view to updating it so it would better support national health 
surveillance. 


2.23 Our follow-up found that there is still no public health legislation and 
that the legislative review is still under way. However, Health Canada has 
made progress in establishing a framework for national health surveillance to 
ensure collaboration among the partners. It has established a number of 
federal/provincial/territorial committees to advise and act on issues of 
national health surveillance (Exhibit 2.1). These committees are a useful way 
to ensure that all the partners participate. The committees have set goals for 
themselves and are working toward them. 
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Exhibit 2.1 Federal/provincial/territorial committees on national health surveillance 


Health Surveillance Working Group—provides a pan-Canadian forum that links the 
federal, provincial, and territorial governments to build a national health surveillance 
infostructure system. The Working Group reports to the Advisory Committee on Health 
Infostructure and the Conference of Deputy Ministers of Health. It has three sub- 
groups: 


* Communicable Disease Surveillance Sub-Group—to provide leadership and 
develop a strategy for the co-ordination and integration of communicable disease 
surveillance systems in Canada. 


¢ Chronic Non-Communicable Disease Infostructure Sub-Group—to provide 
leadership and develop a strategy for the co-ordination and integration of chronic 
disease surveillance systems in Canada. 


Injury Surveillance Sub-Group—to provide advice and recommendations to Health 
Canada, federal/provincial/territorial stakeholder groups, and other organizations 
that maintain injury information systems and to support the effort to build and 
maintain a national injury surveillance infostructure for Canada. 


Canadian Public Health Laboratory Forum—to provide leadership in the public health 
laboratory functions and provide advice on issues related to delivery of services by 
public health laboratories. 


National Health Surveillance Infostructure Project—to develop infostructure systems 
to address issues and gaps related to national health surveillance. 


Canadian Integrated Public Health Surveillance Project Collaborative—to lead in 
defining, developing, and promoting the Canadian Integrated Public Health 
Surveillance Project. 


2.24 As part of its reorganization in 2000, Health Canada created the 
Centre for Surveillance Coordination. Its function is to provide leadership on 
co-ordination issues related to national health surveillance and to support 
the federal/provincial/territorial committees by carrying out specific tasks for 
them. 


An established approach to national health surveillance is still many years away 


2.25 ‘To advance a national framework, the Health Surveillance Working 
Group has agreed on an approach for improving national health surveillance. 
For example, it has agreed that developing a national health surveillance 
“infostructure” system should be a goal of the partners. Subgroups are 
working to develop specific approaches to improving surveillance systems for 
communicable disease, chronic disease, and injury. They are identifying gaps 
in surveillance systems, such as the lack of standards for data, and are 
developing plans to address these gaps. 


2.26 These are comprehensive approaches that the committees believe will 
address many of the weaknesses in national health surveillance. We are 
concerned, however, that Health Canada has set no specific timelines for 
implementing them and so we do not know when the Department expects to 
have the approaches in place. We were informed that given their 
comprehensiveness, it will likely take several to many years. 
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2.27 Also of concern is that these approaches are being developed largely 
independent of current surveillance activities. Rather than incremental 
improvements to current surveillance activities, they involve comprehensive 
change. In the meantime, many of the same weaknesses and gaps continue in 
national health surveillance as we observed in 1999. 


Lack of agreement on roles and responsibilities 


2.28 Clearly defined and agreed-on roles and responsibilities are a key 
element of successful national health surveillance. They are particularly 
important in the areas of collecting and disseminating data and responding to 
public health threats. 


2.29 Lack of agreement on data sharing between Health Canada and the 
provinces and territories. Disease information is the property of the 
provinces and territories. To ensure that this information is shared 
appropriately and that the Privacy Act is not violated, the details of data 
sharing need to be outlined clearly in written agreements. Agreements on 
data collection need to cover such details as how the data will be used, who 
owns the data, what standards will be followed, and how privacy and 
confidentiality will be protected. Agreements on data dissemination need to 
cover such details as what information can be published and who can receive 
it. Finally, each agreement should outline the consequences of not respecting 
ile: 


2.30 At present, only a few agreements on data sharing exist (for example, 
on HIV/AIDS), and no generic agreement has been developed to ensure that 
all important details are covered. Since much of Health Canada’s disease 
information comes from other partners, any agreements would need to clearly 
outline the responsibilities of all partners in the sharing of that information. 


2.31 Health Canada slow to develop common standards for data to be 
shared. We recommended in 1999 that Health Canada establish common 
standards and protocols for classifying, collecting, and reporting data on 
communicable diseases. 


2.32. Common or uniform standards and protocols are critical to ensuring 
that disease information is consistent. Consistency is important because 
national health surveillance involves integrating information so it can be 
analyzed on a national basis. Our follow-up found only limited progress on the 
development of common standards. The Communicable Disease Surveillance 
Sub-Group has begun developing standards for nationally reportable diseases, 
immunization information, and vaccine-associated adverse events (bad 
reactions to a vaccine). Progress has been made on the development of 
standards for data elements and the core data set (the set of data elements 
that are common to all diseases—for example, gender, and date of onset of 
illness). However, only very limited progress has been made on elaborating 
disease-specific data sets (for example, defining the symptoms of a specific 
disease) and laboratory standards (such as which lab test to use). 


233 Once standards have been developed, agreement on them must be 
reached, We found that there is no national agreement on a mechanism for 
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maintaining or approving standards on behalf of all the partners. Without this 
mechanism, Health Canada has no way of ensuring that common standards 
are respected. 


2.34 Still no formal agreement on which diseases should be reported 
nationally. In 1999 we noted that there was no formal agreement on which 
communicable diseases would be reportable (notifiable) to Health Canada 
(those of particular concern, such as tuberculosis and rabies). Nor was there 
agreement on which of several emerging diseases should be added to the list. 


2.35 Our follow-up found that there is still no formal agreement requiring 
provinces and territories to accept or respect the list of nationally reportable 
diseases. However, informal agreement exists on a list of reportable diseases. 
The list was updated in 2000, and work is under way to update it again to 
include several bioterrorist agents, such as smallpox and anthrax. 


2.36 Progress made in agreeing on a co-ordinated response to public 
health threats. Our 1999 report observed that Health Canada had not 
defined clearly who was responsible for doing what in the event of an 
emergency public health threat—for example, controlling diseases at ports of 
entry, managing outbreaks of food-borne disease, or dealing with an influenza 
pandemic. 


2.37 We found that a number of important improvements in this area have 
been made over the last three years or are nearing completion. At this 
writing, a memorandum of understanding (MOU) on controlling diseases at 
ports of entry was soon to be signed by Health Canada and the Canada 
Customs and Revenue Agency. This MOU would provide for training front- 
line customs officers to handle situations that potentially involve the 
importation of illness. Health Canada and the Canadian Food Inspection 
Agency have signed a Food-borne Illness Outbreak Response Protocol and an 
MOU; at this writing, the provincial and territorial governments were being 
consulted in the hope that they would sign the agreements, too. To prepare 
for the possibility of an influenza pandemic, Health Canada has contracted 
privately with a company to ensure that there is an infrastructure to provide 
enough vaccine to make Canada self-sufficient within the next few years. The 
Department is negotiating a new MOU with provincial and territorial 
governments that outlines clearly their roles and responsibilities in the event 
of an influenza pandemic. The MOU will replace an outdated working 
agreement. 


2.38 In summary, Health Canada has made progress on developing a 
framework for national health surveillance. However, it has made only limited 
progress on resolving weaknesses and gaps in national health surveillance 
activities. For the most part, there is still no agreement on important matters 
such as data sharing, common standards, and nationally reportable diseases. 
Strong leadership is needed to ensure that all partners work together to 
advance these initiatives without delay. 
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2.39 Recommendation. Health Canada should work with the provinces 
and territories to set specific timelines for implementing a national approach 
to health surveillance that will ensure that weaknesses and gaps in health 
surveillance are remedied. 


Health Canada’s response. A national approach to health surveillance is set 
out in the Canadian Health Infostructure Health Surveillance Tactical Plan. 
Health Canada and the provinces and territories are supportive of the 
directions set out in this Plan. At the first opportunity, Health Canada will 
discuss with the provinces and territories the setting of time lines under the 


Plan. 


2.40 Recommendation. Health Canada should work with provinces and 
territories to obtain agreement on the sharing of disease information, 
including agreement on data collection, data dissemination, data standards, 
and the list of diseases that should be reported nationally. Further, it should 
work with the provinces and territories to create a mechanism for 
maintaining and accepting data standards. 


Health Canada’s response. Health Canada will continue its work with the 
provinces and territories to obtain further agreements on the sharing of 
disease information, including agreement on data collection, data 
dissemination, data standards, and the list of diseases that should be reported 
nationally. Health Canada will also continue its work with the provinces and 
territories to create a mechanism for maintaining and accepting data 
standards. 


241. In 1999-2000, Health Canada prepared a draft business plan for its 
health surveillance activities (at that time delivered by the Laboratory Centre 
for Disease Control). The development of this plan was a comprehensive 
process and included consultation with leading medical and technical public 
health professionals. Out of this process, Health Canada set its priorities. 
Although this plan was never finalized, it was presented to the Deputy 
Minister of Health Canada, the federal/provincial/territorial Public Health 
Working Group, and the Prime Minister’s Office. 


2.42 Shortly thereafter, in 2000, Health Canada reorganized. The 
Population and Public Health Branch (which then became responsible for 
health surveillance) began a new priority-setting process. Our Office reported 
on this process in our December 2001 Report, Chapter 9. In that report, we 
noted that first steps had been taken to set priorities, including a program 
review exercise. During our follow-up, we determined that Health Canada 
had never completed that exercise. 


2.43 This means that the Department has still not formally established 
priorities among its health surveillance activities. 


2.44 Disease surveillance systems are tools that enable Health Canada to 
track and forecast diseases and injuries, risk determinants, and health 
outcomes. Such systems entail the ongoing collection of disease information; 
the integration, analysis, and interpretation of that information to create 
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surveillance products; and the dissemination of those products to 
governments and public health professionals. Surveillance products can range 
from data such as the national incidence of a disease in a given week toa 
comprehensive annual report that includes incidence, mortality, risk 
determinants, and health care costs for a particular disease. 


Communicable disease surveillance: Improvement has been inconsistent 


2.45 Information on communicable (infectious) diseases comes from a 

number of sources, including physicians, hospitals, public health workers in 
regional health authorities (RHA) or public health units (PHU), and public 
and private laboratories. Exhibit 2.2 illustrates how information should flow 


to Health Canada. 


Exhibit 2.2 How information on communicable disease should reach Health Canada 


Individual feels ill 


Individual seeks medical care 
from 

° physician's office 

* hospital 

¢ public health professional 


Physician makes clinical 


diagnosis of communicable 


Physician informs RHA/PHU informs Province informs 
RHA/PHU* Province Health Canada 


disease 


Sample is sent to lab 


RHA/PHU takes 
appropriate action, 
such as declares 


Province takes Health Canada 
appropriate action, takes appropriate 
such as declares action, such as 


outbreak outbreak declares outbreak 
A 
Lab informs RHA/PHU informs Province informs 
ie Lab makes diagnosis -— RHA/PHU Se Province Health Canada 


Physician 
Lab informs physician =. immediately 


*RHA—Regional health authorities 
PHU—Public health units 


informs individual 
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2.46 Because of the number of sources of disease information (which are all 
outside Health Canada) and the number of transfers of that information 
before it reaches Health Canada, it is a challenge for the Department to 
determine that the information is timely, accurate, and complete. Any 
breakdown in the flow of information affects the information Health Canada 
receives, which in turn affects its ability to take action. 


2.47 Health Canada also needs to ensure that the surveillance products it 
disseminates are timely and contain accurate and complete information if 
they are to be of value. Any deficiencies in the information it receives will 
severely impair the effectiveness of its surveillance products. 


248 In 1999 we reviewed a number of surveillance systems for specific 
communicable diseases. We raised several concerns about the timeliness and 
quality of communicable disease information collected by Health Canada and 
disseminated as surveillance products. The concerns we had about four of 
these systems—the Notifiable Diseases Reporting System and the AIDS, 
influenza, and enteric diseases systems—are summarized in Exhibit 2.3. 


249 Since 1999, the AIDS, influenza, and enteric diseases systems have 
received additional funding, directed by the Treasury Board. Our 2002 follow- 
up of these systems found that Health Canada has improved them. The 
Notifiable Diseases Reporting System did not receive additional funding. We 
found that improvements to this system were limited. In all four systems we 
found more timely dissemination of surveillance products, although the 


Exhibit 2.3 Progress in surveillance of communicable diseases 


Surveillance system 


Notifiable Diseases 


Underreporting by physicians 


Our observations in 1999 . Follow-up observations in 2002 


Limited improvement since 1999 


Reporting System ¢ Delays in receiving data ¢ Monthly reporting about four months 
¢ Data received from provinces with different behind (some provinces missing); 
frequency, in different formats quarterly reporting of preliminary data; 
annual report for 1999 the most recent 
AIDS e Underreporting by physicians ¢ Improvement: field surveillance officers in 
e Delays in receiving data provinces and some formal data-sharing 
e Data incomplete agreements 
¢ Data not sent by all provinces ¢ Annual report for 2001 the most recent 
¢ Timely dissemination 
Influenza ¢ Sentinel physician reporting incomplete ¢ Improvement in four reporting systems— 


Dissemination problematic sentinel physician; laboratory; provincial 


activity; and international 
Weekly reporting, available the following 
week 


Enteric (gastro-intestinal) 
diseases 


Delays in receiving data 


Improvement in three reporting systems— 
laboratory; outbreak; genetic fingerprint 
Weekly laboratory reporting, available the 
following week; real-time reporting 
available for outbreak and genetic 
fingerprint 
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quality of the products is still limited by the quality of the disease information 
on which they are based. The progress we found in these four systems is also 
summarized in Exhibit 2.3. 


250 Exhibit 2.4 describes two other surveillance systems we reviewed—the 
disease-specific surveillance systems for the incidence and outbreak of 
meningococcal infection and for the incidence of vaccine-associated adverse 
events. The exhibit also discusses our concerns about the timeliness and 
quality of the disease information collected. 


2.51 Health Canada’s challenges in collecting communicable disease 
information continue. They include 


* Timeliness. Data sources submit disease information according to their 
own schedules—for example, immediately, daily, weekly, or monthly. 


* Accuracy. Data sources submit disease information in their own ways— 
for example, some give the date of onset as the date of diagnosis and 
others give the date of first symptom. 


* Completeness. Data sources may submit no information or only partial 
information. 


Steps to improve communicable disease information: The Canadian Integrated Public 
Health Surveillance Project 


252 Health Canada recognizes the weaknesses in the disease information it 
collects, and it has developed the Canadian Integrated Public Health 
Surveillance Project to address some of them. This project comprises a group 
of integrated computer and database applications that will allow public health 
professionals in the provinces to collect and forward information on 
communicable diseases to Health Canada. By standardizing reporting, it will 
make disease information more 


* timely, with case-by-case electronic and real-time data; 
* accurate, with common modules and standards; and 


* complete, with the potential to be used at all public health offices and 
laboratories. 


253 The project is managed by a collaborative group of federal and 
provincial representatives. It is really two information management systems. 
The first, the Public Health Information System, manages epidemiological/ 
clinical data. The second, the Laboratory Data Management System, 
manages laboratory data. Health Canada has made satisfactory progress on 
this project. The Public Health Information System is operational in British 
Columbia and is soon to be piloted in several regional health authorities or 
public health units in seven provinces and territories. The Laboratory Data 
Management System is being used in two federal laboratories. 


2.54 This project has the potential to improve the quality of surveillance 
information. Our concerns remain, however, because full implementation is 
still some time away, the disease information in current systems is still 
inadequate, and limited effort has been made to find an interim solution. We 
have identified several matters that need to be resolved before these systems 
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can succeed. First, compatibility of provincial systems with the national 
system must be assured. Second, data standards have yet to be agreed on. 
Third, obtaining agreement on data sharing, data ownership, and privacy 
protection remains a challenge. Finally, the project will not fully address 
concerns about the completeness of the data. The impact of the two systems 
on the timeliness and quality of communicable disease information may not 
be evident for some time. 


Exhibit 2.4 Meningococcal disease and vaccine-associated adverse events 


Meningococcal disease is serious and sometimes fatal. About one in ten persons who 
develop the disease may die and one in ten who recover will suffer some long-term 
effects, such as deafness. The number of cases of meningococcal disease goes up and 
down in regular cycles, with local and regional outbreaks occurring about once every 
10 to 15 years. Although the disease can occur at any age, the highest risk is among 
children under a year old, with the next-highest risk among teenagers aged 15 to 

19 years. 


There are effective antibiotics for the treatment of meningococcal disease. Vaccines 
against several meningococci types are available and may be used in an outbreak 
situation to provide protection. 


Meningococcal disease is a reportable disease and is therefore reported through the 
Notifiable Disease Reporting System. Health Canada also has an enhanced (disease- 
specific) surveillance system for meningococcal disease. Because the disease is 
preventable with a vaccine, any adverse events (bad reactions) following vaccination 
should be reported through the surveillance system for vaccine-associated adverse 
events. 


The enhanced system is intended to receive case-by-case disease information 
(confirmed cases, with case information linked to lab data) and outbreak information 
electronically from the provinces on a real-time basis. However, reporting by the 
provinces is not always timely and complete. Standards for the required information 
have not been developed. Case information for meningococcal disease is sometimes 
submitted independent of the lab data. Matching at the federal level can be difficult, 
and unmatched data have less value. During an outbreak, the quality of data generally 
improves as provinces focus more on meningococcal disease. However, during an 
outbreak the data often arrive through informal channels, such as emergency phone 
calls or e-mails, rather than the formal surveillance system. As a result, the 
communication of timely, accurate, and complete information on meningococcal 
disease is a concern. The most recent report on meningococcal disease was published 
in 2000 and relied on data from 1997 and 1998. 


One means of dealing with a meningococcal outbreak is to vaccinate vulnerable 
population groups. With every vaccination there is a risk of an adverse event. Health 
Canada has two surveillance systems that provide some information on vaccine- 
associated adverse events. The systems are of limited value in collecting timely, 
accurate, and complete information on vaccine-associated adverse events from 
meningococcal vaccination. The first is a network of 12 pediatric hospitals that provide 
information on patients who may be suffering from a serious reaction to the vaccine. 
The system’s value is limited because it would not capture an adverse event unless the 
individual went to a pediatric hospital, and people in one of the most vulnerable age 
groups—15 to 19—are not likely to be patients in a pediatric hospital. The second 
system collects information from public health professionals on a Standard form; it is a 
voluntary system. Health Canada recognizes the limitations: a voluntary system is 
unlikely to provide complete information. However, it believes that the more serious the 
adverse event, the more likely it is to be reported. The last annual report on adverse 
events was published in 1997. 
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255 In summary, Health Canada has made progress in enhancing the 
surveillance systems for which it received directed funding from the Treasury 
Board. It has made only limited progress on the others. The Canadian 
Integrated Public Health Surveillance Project has the potential to address 
some weaknesses, but it still faces some challenges. In the meantime, current 
disease information remains inadequate. 


2.56 Recommendation. Health Canada should enhance its current 
surveillance of communicable diseases while it continues to advance the 
Canadian Integrated Public Health Surveillance Project. 


Health Canada’s response. Health Canada will continue to enhance current 
surveillance of communicable diseases, with particular emphasis on specific 
diseases such as HIV, enteric diseases, sexually transmitted infections, blood- 
borne pathogens, and vaccine-preventable diseases. The Department will 
continue to advance the Canadian Integrated Public Health Surveillance 
Project. 


Chronic disease surveillance: Limited progress 


2.57. Chronic disease information is collected from a number of sources, 
including disease registries, such as those for cancer; administrative data such 
as hospital records and provincial billing records for health services; morbidity 
and mortality data maintained by Statistics Canada; and health surveys such 
as the National Population Health Survey. This information is not sent 
automatically to Health Canada, although it is available to the Department 
for its surveillance activities. 


2.58 Once the information is collected, Health Canada has a number of 
surveillance systems that integrate, analyze, and interpret disease information 
and create and disseminate surveillance products based on that information. 


259 In 1999 we noted problems in the disease information collected for two 
of several cancer surveillance systems, the Childhood Cancer Treatment and 
Outcome Surveillance System and the National Enhanced Cancer 
Surveillance System. The first system is part of a broader surveillance system, 
the Canadian Childhood Cancer Surveillance and Control Program. We 
found that the Treatment and Outcome system had incomplete data on 
youths between 15 and 19 years of age and that no results of any analysis had 
been reported. Our follow-up determined that no progress had been made 
toward filling the information gaps, but a report on the results of its analysis of 
the data collected from the Childhood Cancer Treatment and Outcome 
Surveillance System is scheduled to be published. Also of concern is the very 
limited activity in surveillance for the late effects, etiology investigation, and 
tissue bank systems—the other systems in the Childhood Cancer Program. 


2.60 In 1999, it was determined that there were gaps in the environmental 
data collected by National Enhanced Cancer Surveillance System. Our 2002 
follow-up found that the information gaps had not yet been filled and that 
more current information has not been collected. Health Canada has 
published a number of reports on the results of its analysis. 
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261 Recommendation. Health Canada should collect the disease 
information needed to provide a complete and timely picture of cancer in 


Canada. 


Health Canada’s response. Health Canada’s current picture of cancer in 
Canada includes cancer incidence and mortality data. Health Canada needs 
more concrete information on risk factors, severity at the time of diagnosis, 
and treatment data. The Department will continue its work with the 
provinces and territories to get such concrete information. 


Gaps in national health surveillance © 262 Most of Health Canada’s existing disease surveillance systems focus on 
activities the incidence of disease. However, there are opportunities for collecting and 
analyzing information on disease beyond its incidence. 


2.63 As an individual moves from health to disease to health outcome, that 
individual will interact with the health care system many times (Exhibit 2.5). 
Each time, disease information can be collected for surveillance activities 


(Exhibit 2.6). 


2.64 Gaps in surveillance exist because there may be no surveillance system 
at all or because an existing surveillance system may focus largely on the 
incidence of disease and not on other important areas. Those areas can 
include risk determinants and the impacts of interventions, screening, and 
treatment on health outcomes. 


2.65 In its 1999-2000 draft business plan, Health Canada recognized a 
number of gaps in surveillance, particularly surveillance of chronic diseases. It 
also demonstrated the significance of chronic diseases to the health of 
Canadians, as measured by the projected cost of these diseases for 1998 


(Exhibit 2.7). 


2.66 In 1999 we noted that national surveillance activity for cardiovascular 
disease was inadequate—specifically, surveillance of incidence, risk 
determinants, intervention, and treatment. Our report acknowledged that 
Health Canada was working to develop a national system that it expected 
would be operating fully in 2003. We also noted a gap in national surveillance 
of diabetes, but a national system was to be operating fully in 2000. 


2.67 Our follow-up found that there are still some significant gaps in 
surveillance: 


* Cardiovascular disease surveillance. Health Canada has done little work 
on developing a national system for surveillance of cardiovascular 
disease, a disease projected to cost Canadians $21 billion in 1998. 
During our 1999 audit, work had begun on developing a Canadian 
Heart Disease and Stroke Surveillance System Network. The Network 
made some initial progress but after a business plan was developed, the 
funding for the Network ended. Health Canada’s present activities in 
this area are directed largely at publishing a biannual report on heart 
disease and stroke that provides only very limited information on the 
disease. The Chronic Non-Communicable Disease Infostructure Sub- 
Group is currently contributing to a study on the feasibility of extending 
the diabetes surveillance system to include cardiovascular disease. 
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Exhibit 2.5 Public health services and disease continuum 
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Exhibit 2.6 Opportunities in the delivery of public health services for 
surveillance 


Opportunities exist in the delivery of health care for 
surveillance systems to monitor the following: 


Risk determinants (behaviour, environment, age, gender, 
social support, etc.)—to identify the factors that influence 
disease. 


Health promotion, health protection, disease prevention— 
to evaluate the effectiveness of these interventions. 


Screening—to evaluate the effectiveness and coverage of 
screening activities. 


Incidence—to quantify how many new cases of a 
particular disease have been identified in Canadians 
during a given period. 


Prevalence—to quantify the total number of cases of a 
particular disease in the Canadian population during a 
given period. 


Outbreak—to identify an outbreak of a communicable 
disease and to plan a response to the outbreak. 


Treatment—to evaluate the effectiveness of a treatment 
and the economic burden of the treatment, for example, 


¢ Drug—to evaluate the effectiveness of drugs and 
adverse reactions to drugs, and to identify drug 
resistance 


Surgical—to evaluate the effectiveness of surgical 
practices and to develop best practices 


¢ Vaccination—to evaluate the effectiveness of 
vaccination strategies, longevity of protection, and 
adverse reactions to vaccines. 


Rehabilitation—to evaluate the effectiveness of 
rehabilitation options and to develop best practices. 


Support (hospital, home care, and palliative care)—to 
evaluate the effectiveness of support and its economic 
burden and to measure accessibility to health care. 


Health—to measure the health of Canadians. 
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Exhibit 2.7 Projected costs of selected diseases for 1998 


Si ($ billions) 
: ; ae : | tren inact Total 
_ Diagnostic category ; cost cost cost 
Catdiovacoutar en heart disease a | 23 37, Poh 
stroke) 
Musculoskeletal (i.e., arthritis) 2.5 125 18 
Injuries Shas 10.9 14 
Cancer Se 9.8 iS 
Chronic Respiratory (i.e., asthma) Sali 8.3 12 
Mental Illness Sed 29) 8 


Source: Health Canada Business Plan, 1999-2000 


* Diabetes surveillance. Health Canada has made significant progress on 
developing a national diabetes surveillance system (Exhibit 2.8). 


¢ Other important chronic diseases. There are still a number of key 
health issues for which surveillance is inadequate. These include 
chronic diseases such as musculoskeletal disease (for example, arthritis), 
cancer (except breast cancer), chronic respiratory disease (for example, 
asthma), and mental illness (for example, depression). These four 
diseases had projected costs of about $50 billion in 1998. 


¢ Risk determinant and health outcome surveillance. There are many 
gaps in the surveillance of risk determinants and the impacts of 
interventions, screening, and treatment on health outcomes. For 
example, while the development of a surveillance system for breast 
cancer screening is progressing, no surveillance system currently collects 
information on patient outcomes after various breast cancer treatments. 
As a result, physicians and breast cancer patients have only limited 
information on treatment options (Exhibit 2.9). Similarly, the diabetes 
surveillance system will be able to provide information on the incidence, 
prevalence, and associated health problems of diabetes but not on risk 
determinants, intervention, treatment, or incidence in children under 
19 years of age. An inventory of opportunities for surveillance systems 
for notifiable diseases, AIDS, cardiovascular disease, and cancer 
(excluding breast cancer) is included in Exhibit 2.10. 
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Exhibit 2.8 National surveillance system for diabetes 


Diabetes is the seventh leading cause of death in Canada. It is estimated that 

33 percent of persons with diabetes do not know they have it. If left untreated or 
managed improperly, diabetes can slowly damage the body. This creates complications 
such as heart disease, blindness, and kidney disease. It is also estimated that 

90 percent of all persons with diabetes suffer from type II diabetes, which can be 
prevented or delayed by modifying two known risk determinants—obesity and physical 
inactivity. 


In 1999, the federal government developed the Canadian Diabetes Strategy, with four 
main components. One component was the development of a national diabetes 
surveillance system, and it has seen significant progress since then. All 13 provinces 
and territories are signatories to a memorandum of understanding that outlines the 
participation and responsibilities of all partners. 


The data for the system are administrative data, such as physicians’ billings and 
hospital discharges. Each year, Health Canada receives these data from the provinces, 
separated between cases that might indicate diabetes and cases that do not. The data 
are used to determine the prevalence of diabetes. 


Health Canada intends to develop mathematical models that can be applied to the 
data to correct for false positives. This would allow it to estimate the incidence of the 
disease. 


In the future, the Department will be able to analyze administrative data to identify 
other health problems (complications) associated with diabetes. 


So far, data on prevalence, incidence, and complications from 1995 to 2000 have 
been collected from nine provinces and territories. The information on prevalence is 
being analyzed and the Department plans to issue its first annual report in 2002. 


Exhibit 2.9 Surveillance system for breast cancer treatment 


Breast cancer is the most common cancer among women in Canada. It is diagnosed in 
more than 19,000 Canadians every year. A study by the National Cancer Institute of 
Canada suggested that the direct costs of breast cancer totalled $7 billion in 1990. 
That is just one third of the estimated indirect costs. 


In 1993 the Canadian Breast Cancer Initiative was launched, with a financial 
commitment of $25 million for five years. In 1998 the program was renewed to 2003, 
with total funding of $35 million or $7 million annually. The goal of the initiative is to 
encourage and support research related to the prevention, treatment, and control of 
breast cancer. The initiative has five components, one of which is surveillance and 
monitoring of breast cancer. This component is divided into three sub-components: 
screening surveillance; risk factor surveillance; and treatment and palliative care 
surveillance. 


It appears that most effort has been directed at surveillance of breast cancer screening. 
A system for collecting data from organized screening programs already exists and is 
soon to be extended to include non-organized screening programs. Health Canada has 
undertaken only limited activities to develop a breast cancer treatment surveillance 
system; at present there is no such system. , 


Surveillance information on breast cancer treatment would be of great value to 
physicians and breast cancer patients. For example, long-term effects of treatment 
could be monitored and used to assist physicians and patients in decision making. This 
represents a missed opportunity. 
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* Injury surveillance. The national system for surveillance of injuries to 
Canadians, which had an estimated annual cost of $14 billion in 1998, is 
incomplete. At present, injury surveillance is largely limited to children. 
The Injury Surveillance Sub-Group has prepared an inventory of data 
sources and surveillance activities on injuries. It has also begun an injury 
surveillance pilot project with the British Columbia Injury Research and 
Prevention Unit, which captures injury data for all ages directly from 
10 hospital emergency departments in British Columbia. 


2.68 In summary, a significant number of gaps remain in the surveillance of 
chronic diseases and injuries. Health Canada’s efforts are limited largely to 
surveillance of diabetes and breast cancer. Consequently, it has only limited 
surveillance information on many of the leading causes of death in Canada. 


2.69 Recommendation. Health Canada, in collaboration with other 
organizations, should take steps to fill identified gaps in surveillance by 
ensuring the timely development of national surveillance systems for chronic 
diseases. 


Health Canada’s response. Health Canada has surveillance systems for 
chronic disease at various levels of maturity and, in collaboration with 
provinces and territories and others, will continue its work toward addressing 
specific gaps such as cardiovascular disease, chronic respiratory disease, and 


musculoskeletal disease. 


Exhibit 2.10 Opportunities for surveillance systems—Four diseases 


fe ; : | | Cancer 

Sa eres Notifiable Cardio- (system excludes 
Surveillance opportunity diseases — AIDS vascular breast cancer) 
Risk fe eronane N/A ‘S) © @ 
deere Te health protection, disease O @ O © 
Screening N/A | N/A O O 
Incidence oo) © eo ) 
Prevalence ] e © O @ 
Outbreak © N/A N/A N/A 
Treatment @ O O O 
Support O e@ O | @ 
Health O O @ C) 


@ Surveillance system exists but has weaknesses, as discussed in the chapter. 


O There is no surveillance system, or surveillance is limited. 


N/A Not applicable. 


Report of the Auditor General of Canada—sSeptember 2002 


Chapter 2 19 


HEALTH CANADA—NATIONAL HEALTH SURVEILLANCE 


e 


oul 


rveillance issues that need to be 


resolved 


270 As we have noted, Health Canada has made limited progress on 
improving national health surveillance. During our follow-up we identified 
several factors that have impeded or prevented improvement. 


2.71 Many of the factors within Health Canada’s control are financial. The 
centres that conduct surveillance activities receive funding for salaries, 
operations and maintenance, and grants and contributions. This funding, 
which represents the centres’ financial capacity to deliver programs, comes 
from three streams. The first stream is referred to as targeted money, funds 
that the Treasury Board has approved for specific initiatives, mainly within 
the last five years. As part of obtaining the approval, the Department agrees 
to achieve certain goals. The Centre for Infectious Disease Prevention and 
Control has received targeted funds for blood-borne pathogens, enteric 
diseases (gastro-intestinal illness), influenza pandemic, HIV/AIDS, and 
hepatitis C. The Centre for Chronic Disease Prevention and Control has 
received targeted funds for diabetes and breast cancer. 


272 The second stream of funding is referred to as A-base money. It is the 
funding that Health Canada allocates to the centres for programs that do not 
have specifically targeted funding. This stream represents the only 
discretionary funding available to the centres. 


273 The third stream is the annual reallocation of resources within Health 
Canada to offset funding pressures on the Department. This exercise, carried 
out by the Department’s Resource Allocation Committee, is discussed in 
more detail in paragraph 2.81. 


274 Funding in 2001-02 to three key centres for operations and 
maintenance of all their activities is outlined in Exhibit 2.11 (The funding 
shown includes surveillance activities as well as the other activities of the 
centres.) 


Exhibit 2.11 Operations and maintenance funding in 2001-02 (at 1 April 2001) 


($000). 
Centre for Infectious | Centre for Chronic . 3 
Diseases Prevention | Diseases Prevention Centre for Surveillance 
Program and Control andControl =| Coordination 

Targeted funds 23,500 7,900 900 
A-base 3,850 1,050 300 
Departmental 
reallocation O 0 6) 
Total funding 
before levies 27,350 8,950 1,200 
Levied amounts 
(deducted) (4,200) (1,500) (200) 
Total after levies 2350 7,450 1,000 
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Lack of financial capacity to establish chronic disease surveillance systems 


275 Health Canada’s 1999-2000 draft business plan also recognized that 
the Department lacked the financial capacity to fill the gaps in disease 
surveillance. It recommended that resources be directed to 


* maintain and extend infectious disease activities, where required; and 


* extend the scope of activities for chronic diseases. 


2.76 However, our follow-up found that the recommendations in the draft 
business plan had not been implemented. We were informed that financial 
capacity remains a concern in 2002. In 2001—02 the Centre for Chronic 
Disease Prevention and Control received $5 million for diabetes ($1 million 
of it for surveillance activities); $2.6 million for breast cancer ($0.6 million for 
surveillance activities) ; $300,000 for biotechnology (none of it for 
surveillance activities); and $1.05 million in A-base funding ($0.6 million for 
surveillance activities). 


2.17 The $600,000 in A-base funding is all that is available for all chronic 
diseases except diabetes and breast cancer—that is, cardiovascular disease, 
other forms of cancer, musculoskeletal disease, chronic respiratory disease, 

and mental illness. 


Lack of financial capacity to maintain chronic disease surveillance systems 


2718 The Childhood Cancer Treatment and Outcome Surveillance System 
and the National Enhanced Cancer Surveillance System both received 
targeted funding for their development. Once they were established, targeted 
funding ended and an amount to maintain the systems was included in the 
A-base. Over time, however, the amount in the A-base has eroded. 


279 The Brighter Futures—Child Development Initiative, for example, 
targeted funds to the development of the Childhood Cancer Treatment and 
Outcome Surveillance System for five years. When the Child Development 
Initiative ended, funding of $1 million to keep the surveillance system 
operating was added to the A-base. However, the A-base funding has 
declined consistently (Exhibit 2.12). 


2.80 The Population and Public Health Branch has reported that this 
funding is inadequate. In 2001-02, the Branch identified this surveillance 
system as one of several surveillance systems that were an unfunded, high- 
priority project of the Centre for Chronic Disease Prevention and Control. 
The Branch asked the Department’s Resource Allocation Committee for 
additional funding of $420,000 to operate this system. However, the Centre 
received no additional funding and had to manage the system within its 
A-base resources, which we have noted totalled $600,000 for all chronic 
diseases except diabetes and breast cancer. 


Erosion of financial capacity 


281 Health Canada engages each year in an internal exercise of resource 
reallocation, managed by its Resource Allocation Committee. The 
Committee collects money by charging levies to the funds allocated to each 
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branch of the Department. Inadequately funded activities of all the branches 
are identified, and those with highest priority are allocated additional funds 
from the money collected through the levies. The levies charged to the 
Population and Public Health Branch are collected from the operations and 
maintenance funding for each of the Branch’s centres. 


282 Other levies to fund various initiatives of the Department are also 
collected from the centres. Our analysis of resources allocated to each centre 
determined that the levies charged to the Branch for 2001-02, and then 
collected from each centre as a percentage of its operations and maintenance 
funding, were roughly equivalent to 


* 15 percent from the Centre for Infectious Disease Prevention and 
Control, 


* 17 percent from the Centre for Chronic Disease Prevention and 
Control, and 


* 17 percent from the Centre for Surveillance Coordination. 


The three centres paid a total of almost $5.9 million in levies (Exhibit 2.11), 
taken from both the targeted funds intended for specific activities and the A- 
base funds, which are discretionary. The funds were reallocated to other 
activities within Health Canada, which means they are supporting activities 
that are not the ones to which the Treasury Board directed the support. 


2.83 In 2001-02, the three centres also identified unfunded activities of 
about $7 million. The Branch requested additional funding for these activities 
from the Resource Allocation Committee. However, the Committee denied 
the requests and provided no additional funds. The collection of levies, 
combined with the denial of requests for funding, has eroded the centres’ 
ability to carry out their surveillance activities. 


Exhibit 2.12 A-base funding for Childhood Cancer Treatment and Outcome Surveillance, 1996-2002 


Funding ($ thousands) 
1,200 


1996-97 1997-98 1998-99 1999-00 2000-01 2001-02 
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Priorities need to be established and supported with funding 


2.84 A great many diseases affect Canadians; there are many opportunities 
for health surveillance. It is clear that not all of the information on diseases 
can be collected and analyzed. 


285 It is essential that Health Canada determine its priorities for national 
health surveillance activities relative to the other activities in the 
Department. As well, priorities among the various health surveillance 
activities must be clearly established. This means choosing which key health 
issues will be surveyed—for example, outbreaks of reportable (notifiable) 
diseases or interventions for cardiovascular disease. Finally, it is important 
that Health Canada direct enough resources to those priorities to develop 
surveillance systems and maintain them. 


2.86 As already noted, Health Canada did set priorities for health 
surveillance activities in 1999-2000. However, those priorities were not 
supported by funding. And we have already noted that more recent attempts 
to set priorities have not been completed. 


2.87 Health Canada has made only a limited commitment of funding to the 
centres that conduct national health surveillance. We determined that 
around 86 percent of the operations and maintenance funding received by 
the Centre for Infectious Disease Prevention and Control is targeted funding. 
And 89 percent received by the Centre for Chronic Disease Prevention and 
Control is targeted. This means that most of the centres’ activities are 
supported by funding directed to the centres by the Treasury Board. Of the 
funds the two centres receive for their activities, only 14 percent and 

11 percent, respectively, come from Health Canada. 


288 These financial capacity issues create a number of problems for 
program officials. Funding is variable, the A-base is eroded, and targeted 
funds are redirected, although the centres remain responsible for delivering 
adequate national health surveillance. 


289 Recommendation. Health Canada should identify its health 
surveillance priorities and ensure that adequate and stable funding is 
available to develop and maintain the surveillance systems that it identifies as 
priorities. 


Health Canada’s response. For Health Canada, health surveillance activity 
is of significant importance. It is a constant and important item for analysis 
and evaluation at the time of the annual setting of departmental priorities, 
and it will remain so in the future. 
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Keeping Canadians safe: Food-borne 


disease and blood-borne pathogens 


2.90 In response to blood and food crises, the Treasury Board directed 
specific funds to these program areas so that surveillance systems could be 
improved. We assessed Health Canada’s progress in improving the 
surveillance systems for food-borne disease and blood-borne pathogens. 


National surveillance of enteric diseases 


291 Since 1999, the Department has made progress on developing systems 
for national surveillance of enteric diseases. Enteric diseases cause 
gastrointestinal illness and are usually transmitted in food and water. Health 
Canada has undertaken five major initiatives to improve its surveillance 
systems: 


* National Enteric Surveillance Program. A system that collects and 
analyzes all positive laboratory samples from 10 provincial laboratories. 
The results are compared against a historical baseline to detect any 
unusual increase in the incidence of pathogens. 


¢ Canadian Enteric Outbreak Surveillance Centre. Two linked Web- 
based applications that allow information on outbreaks to be shared in 
confidence among regional, provincial/territorial, and federal public 
health professionals on a real-time basis once they are posted on the 


Web. 


* PulseNet North. A laboratory outbreak identification system that 
allows information on pathogens to be exchanged in real time. It is 
linked with the provincial labs in all provinces in Canada and with a 
similar system in the United States (PulseNet), allowing information 
sharing across North America. 


¢ National Studies on Acute Gastrointestinal Illness. A collection of 
studies to supplement existing surveillance activities, for example, 
studies to determine the underreporting of gastrointestinal illness and 
identify the risk determinants associated with gastrointestinal illness in 


Canada. 


* National Enteric Surveillance Program Stakeholder Committee. A 
federal/provincial/territorial committee to advise on and discuss issues 
related to national surveillance of enteric diseases. 


2.92 The National Enteric Surveillance Program and PulseNet North have 
been implemented. The Canadian Enteric Outbreak Surveillance Centre 
finished its pilot phase and is being rolled out in Ontario in 2002. The other 
initiatives are still quite new but appear promising. Health Canada has also 
identified areas that need more work, and it plans to make the necessary 
improvements. 


National surveillance of blood-borne pathogens 


2.93 In 1996, while the Krever Commission was still finalizing its report on 
blood safety, the Treasury Board approved funding to strengthen Health 
Canada’s Blood Safety Program. In 1998, after the Commission issued its 
report, the Treasury Board approved additional funding for the Program. A 
portion of the funding was to be used to develop the capacity to create a 
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blood surveillance system. The Department committed to have such a system 
in place by 2003. 


2.94 Health Canada has made progress on developing a national system for 
the surveillance of blood-borne pathogens. It initiated the Transfusion- 
Transmitted Injury pilot surveillance system in 1998, modelled on the Quebec 
system that was already being developed. The Quebec system is based on a 
network of transfusion safety officers who collect data from hospitals that 
have a significant level of transfusion activity. The pilot started with Quebec 
and British Columbia; Nova Scotia and Prince Edward Island were added in 
2000 and they submitted data in 2002. Ontario and Manitoba joined in 2002. 


2.95 During our follow-up, however, we became aware of issues yet to be 
resolved that could delay the implementation of the National Transfusion- 
Transmitted Injury surveillance system. Specifically, we are concerned that 
Health Canada has not made adequate progress toward assuring that timely, 
accurate, and complete information from all provinces will be available in 
2003. Nor can it be sure that provincial capacity is adequate and can be 
maintained to provide the information needed for a sustainable national 
system. 


2.96 We will continue to monitor the progress of this surveillance system. 


Measuring and reporting 297 In 1999 we noted that Health Canada had completed few evaluations 
performance and had no formal plan to evaluate surveillance systems. We also noted that 
the framework for measuring performance and the development of 
performance indicators were incomplete. Finally, we noted that the 
Department was reporting only limited information to Parliament. 


298 During our 2002 follow-up, we noted that the Departmental Program 
Evaluation Division was working on evaluations of the Canadian Strategy on 
HIV/AIDS and Health Canada’s Blood Safety Program. The Population and 
Public Health Branch has also evaluated a few surveillance systems. This is 
progress; but given that three years have passed, it is too slow. 


299 The Department is still developing its performance measures. They 
have not been approved by senior management, nor have they been 
distributed widely in the Department. 


2.100 Finally, reporting to Parliament remains inadequate. It is limited largely 
to information on activities and outputs. It does not include information on 
issues such as the value of surveillance information to support evidence-based 


decisions. 


2.101 Recommendation. Health Canada should strengthen its evaluation, 
performance measurement, and reporting of results of its health surveillance 


activities. 


Health Canada’s response. The Department has strengthened its evaluation 
and performance measurement of health surveillance. Examples include the 
public release of the health indicators of First Nations health status and the 
enhanced use of performance information in the departmental performance 
report. In the performance report, performance expectations are made clear 
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Standing Committee on Public 


Accounts 


and concrete, and in the vast majority of cases, key results are reported 
against expectations. The reliability of performance information is supported 
by facts, and the use of performance information is demonstrated. Progress 
will continue to be made in the future. 


2.102 On 2 March 2000, the Standing Committee on Public Accounts 
invited our Office and Health Canada to a hearing on Chapter 14 of our 
September 1999 Report. In May 2000, the Committee released a report that 
included five recommendations. 


2.103 The recommendations focussed on two themes. Four recommended 
improvements in reporting to Parliament on specific activities 
(implementation of the Network for Health Surveillance, results and 
outcomes of the Network, performance measurement and risk assessment, 
and implementation of the recommendations in our Report). The fifth 
recommended the development of tools to assist in evaluation, performance 
measurement, and risk assessment. 


2.104 We found that Health Canada has reported some information about 
the activities of the Network for Health Surveillance. It provided information 
on important issues such as strategic plans, timelines, and budgets, but the 
progress it discussed was not measured against those plans, timelines, and 
budgets. As already noted, reporting on performance measurement and risk 
assessment remains limited, and only a few evaluations have been completed. 
The Department provided a status report in 2000 on its implementation of 
our recommendations, but not in 2001. Finally, we found in 2002 that Health 
Canada is still developing the tools it needs to assist in evaluation, 
performance measurement, and risk assessment. 


Conclusion 


2.105 Since our audit in 1999, Health Canada has made progress in 
establishing a national framework that allows for greater collaboration among 
the partners involved in health surveillance. It has also made progress in 
defining roles and responsibilities for responding to public health threats. 
However, for the most part, it still has no agreement with the other partners 
on important matters such as data sharing, common standards, and nationally 
reportable diseases. 


2.106 Many of Health Canada’s existing disease surveillance systems still lack 
timely, accurate, and complete disease information, although some 
surveillance systems have improved. To address some of the weaknesses in the 
surveillance of communicable diseases, Health Canada and the provinces 
have undertaken to co-operate on developing the Canadian Integrated 
Public Health Information Surveillance Project. They have made progress, 
but full implementation is not likely for some time. 


2.107 For the most part, diabetes and breast cancer are the only chronic 
diseases with a national surveillance system. The Department has allocated 
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only limited resources to the surveillance of other chronic diseases, despite 
the recognized importance of this activity, and it lacks health surveillance 
information on cardiovascular disease, cancer (except breast cancer), 
musculoskeletal disease, chronic respiratory disease, mental illness, and 
injuries. Further, it is lacking in surveillance of risk determinants and the 
impact of interventions, screening, and treatment on health outcomes. 


2.108 We identified several issues that Health Canada needs to resolve. At 
present, these issues are impeding or preventing progress. Many of them are 
financial issues. The Department lacks the financial capacity to fill gaps in the 
surveillance of chronic disease. It also lacks the financial capacity to maintain 
chronic disease surveillance systems. Further, the internal exercise of resource 
reallocation results in an erosion of financial capacity. Finally, Health Canada 
needs to set priorities for health surveillance activities and provide adequate 
funding to support those priorities. 


2.109 Health Canada’s evaluation, performance measurement, and reporting 
of results of its health surveillance activities remain weak. 


2.110 We recognize that Health Canada is not solely responsible for a number 
of these weaknesses. We encourage it to continue to show strong leadership 
for collaboration and co-operation among all the partners in health 
surveillance to address these problems. 
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About the Follow-Up 


Objective 
The objective of this audit was to determine the progress made by Health Canada in developing the following: 


* common standards and protocols to facilitate collaboration and sharing of information among all the players in 
the health surveillance process; 


* means to ensure that required surveillance systems provide timely and relevant information to anticipate, 
prevent, and respond to health threats and emerging health risks; 


* adequate procedures to measure performance and report results; 


* solutions to identified problems. 
Scope 


The follow-up audit focussed on the recommendations made in our 1999 Report—Chapter 14, National Health 
Surveillance: Diseases and Injuries; and Chapter 15, Management of a Food-borne Disease Outbreak. We also 
followed up on recommendations made by the Standing Committee on Public Accounts. Health Canada has 
undertaken a number of new initiatives in the last three years, which we also reviewed. 


We reviewed a departmental status report on the action taken in response to the recommendations. We carried out 
extensive interviews with Health Canada staff involved in surveillance activities. We also met with provincial health 
officials in several provinces and with local medical officers of health in several regions. Finally, we reviewed 
documentation supporting Health Canada’s activities in health surveillance. 


Our report presents the results of the areas that were re-audited. 


Criteria 


We expected that Health Canada would have made satisfactory progress in implementing our recommendations and 
those of the Public Accounts Committee. 


The criteria from the 1999 audits remain relevant. Therefore, we expected that 


¢ all players in health surveillance would clearly understand Health Canada’s role and responsibilities in the 
national health surveillance process; 


* a full range of documented protocols and procedures would exist that indicate clearly what each player should 
do when a disease outbreak or health threat occurs; 


* all players in the health surveillance process would use common standards and protocols for classifying, 
collecting, and reporting data on diseases and injuries to ensure that all information is comparable across the 
country; 


* Health Canada’s surveillance systems for monitoring diseases and injuries would enable it to collect, analyze, 
and disseminate all the information needed to help anticipate, prevent, and respond to existing and emerging 


health risk; 


* there would be procedures to measure the effectiveness of health surveillance activities and report results; 


* Health Canada’s health surveillance activities would be based on a sound risk-benefit approach and a rational 
priority-setting framework; and 


* Health Canada would take appropriate action to resolve identified problems. 
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Ratings 


We assessed the action of departments/agencies against our original audit recommendations (see Key Message at the 
beginning of the Chapter). We used the following ratings: 


Completed. Corrective action has been fully implemented 
Satisfactory progress. Progress is being made at a satisfactory pace 
Limited progress. Some progress is being made, but the pace or scope is not satisfactory 


No progress. No evidence of progress although the department or agency accepted the recommendation from 
the original audit 


Rejected. The department or agency did not accept the recommendation from the original audit 


Unknown. Status of progress is unknown or information is not available 


Audit team 


Assistant Auditor General: Maria Barrados 
Principal: Patricia MacDonald 
Director: Linda Anglin 


Jo Ann Little 
Marie-Eve Viau 


For information, please contact Communications at (613) 995-3708 or 1-888-761-5953 (toll-free). 
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Health Canada 
Federal Support of Health Care Delivery 


The audit work reported in this chapter was conducted in accordance with the legislative mandate, policies, and practices of the 
Office of the Auditor General of Canada. These policies and practices embrace the standards recommended by the Canadian 
Institute of Chartered Accountants. 
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Increased monitoring capacity 

Limited number of information sources for monitoring 

Limited access to the information that could identify non-compliance 

Health Canada still does not report the extent of compliance 

First Ministers agree to report publicly on performance of their health care systems 
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Health Canada 
Federal Support of Health Care Delivery 


Assistant Auditor General: Maria Barrados 
Principal: Patricia MacDonald 


Key Message 


3.1 Health Canada has made only limited progress in addressing the weaknesses we identified in our 1999 audit. 
As a result, its monitoring still does not allow it to assess and report the extent of provincial and territorial 
compliance with the Canada Health Act. Resolving disputes over compliance with the Act remains slow. Both levels 
of government recently agreed to a process for dispute avoidance and resolution that holds potential for improving 
the resolution of disputes through co-operation and collaboration. The federal government still does not identify its 
intended contribution to health care funding. Parliament and Canadians need that information for informed debate 
on the future of health care. 


ORIGINAL ISSUES STATUS RATING* 


3.2. Health Canada should assess the capacity of the 
information sources it uses for monitoring the operation 
of the Canada Health Act and determining the extent to 
which provinces and territories have satisfied the Act’s 
criteria and conditions. 


Health Canada has increased its staff and budget to 
monitor and assess compliance with the Act. It has also 
developed a process for consistent, proactive monitoring 
and improved its information systems. Health Canada 
: LIMITED 

needs to continue to work with the provinces and 

aa ae PROGRESS 
territories because it still does not have adequate 
information to assess the extent of provincial and 
territorial compliance with the Canada Health Act 
criteria and conditions for health care funding. 


The Canada Health Act Annual Report provides a good 
description of provincial and territorial health care 
insurance legislation, as well as statistics on health care 

: ee LIMITED 
delivery. The Report does not indicate the extent to PROGRESS 
which each provincial and territorial health care 
insurance plan has satisfied the Canada Health Act 
criteria and conditions. 


3.3. In its annual reports to Parliament, Health Canada 
should clearly indicate the extent to which each 
provincial and territorial health care insurance plan has 
satisfied the Canada Health Act criteria and conditions. 
Where it does not provide this information in the reports, 
it should clearly explain the reasons. 


3.4 The federal government should explore options to 
improve information on its total contribution to provinces 
and territories for health care. 


The federal government provides only limited information 

aa Eiceee. F LIMITED 
on its intended total contribution to the provinces and PROGRESS 
territories for future funding of health care. 


* Possible ratings are completed, satisfactory progress, limited progress, no progress, rejected, unknown. (See About the Follow-Up for an explanation of the ratings.) 


Health Canada and the federal government have responded. Health Canada has not agreed with the 
recommendations but will address any gaps with regard to information collection in order to fulfill its obligations to 
administer the Canada Health Act. Health Canada has committed to further improving its performance 
measurement and reporting. The federal government has agreed to continue to explore options to improve health 


care information. The responses are included in the chapter. 
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Introduction 


The federal government is a significant player in health care 


35 The federal government is not directly responsible for the delivery of 
health care services in the provinces and territories; that is a provincial and 
territorial responsibility. However, the federal government supports health 
care by transferring funds to the provinces and territories to assist them in 
carrying out their health care mandates. 


36 The federal government funds health care in a variety of ways. It 
provides direct funding for research, surveillance, and the development of the 
health “infostructure.” It also funds grants and contributions to individuals 
and organizations to participate in activities such as health promotion, health 
protection, disease prevention, and health research. The federal government 
does deliver health care services directly to specific groups of people, such as 
First Nations and Inuit, the Canadian Forces, veterans, inmates of federal 
penitentiaries, and members of the Royal Canadian Mounted Police. 


3.7 The largest federal transfer to the provinces and territories provides 
support for health care, post-secondary education, and social assistance. 
These transfers and the other forms of health care funding make the federal 
government a significant player in health care. 


Canada has a long history of publicly financed health care 


38 Until the late 1940s, health care in Canada was dominated by private 
medicine and access to care was based on ability to pay for it. In 1947, 
Saskatchewan introduced a public insurance plan for hospital services that 
covered all residents, regardless of their ability to pay. This began the 
evolution of Canadian health care into the system we have today, a system 
that is publicly funded and aims to ensure that all residents of Canada have 
prepaid access to the health care they need. Exhibit 3.1 summarizes the key 
milestones in this evolution. 


39 In 1957, the federal government introduced the Hospital Insurance and 
Diagnostic Services Act in an attempt to encourage all provinces to develop 
hospital insurance plans. Under the Act, the federal government offered to 
share the costs of eligible services roughly fifty-fifty with the provinces. As a 
condition for receiving federal money, the provinces and territories agreed to 
make insured services available to all of their residents on uniform terms and 
conditions. By 1961, all 10 provinces and the two territories had signed 
agreements establishing public insurance plans that provided universal 
coverage for in-patient hospital care. 


3.10 In 1966 the government introduced the Medical Care Act, which 
provided for federal funding to cover close to half the cost of physician visits 
and services. To qualify for federal funding, a province or territory had to 
ensure that its medical insurance plan satisfied four criteria: it had to be 
publicly administered, portable, and universal; and insured services had to be 
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accessible. By 1972, all provinces and territories had extended their health 
insurance plans to include physician services. 


3.11 The Canada Assistance Plan was also introduced in 1966. This was a 
federal—provincial program for sharing the costs of comprehensive welfare 
services. In 1977, the federal government established the Extended Health 
Care Services Program to provide financial assistance to the provinces and 
territories for ambulatory care, nursing home intermediate care, adult 
residential care, and home health care. 


3.12 In 1977, the federal government replaced cost sharing with block 
funding. This move responded to concerns about the expense and 
unpredictability of cost sharing and allowed more flexibility for the provinces 
to set their spending priorities. A new mechanism for block fund transfers was 
introduced, called Established Programs Financing (EPF). The EPF combined 
federal transfers for hospital and medical services with transfers for post- 
secondary education and the Extended Health Care Services Program. 


3.13. The federal health transfer through Established Programs Financing 
was in roughly equal portions of cash and tax point transfers. To provide the 
tax transfer, the federal government reduced its personal and corporate 
income tax rates, which allowed provinces to raise their tax rates by an equal 
amount. As a result, the revenue that would have flowed to the federal 


Exhibit 3.1 Key milestones in the evolution of universal, publicly financed health care in Canada 


1947. | Saskatchewan introduced a public insurance plan for hospital services. 


1957 The federal government introduced the Hospital Insurance and Diagnostic Services Act, a cost-shared program 
providing insurance coverage and access to hospital services. 


1958-61 | Provinces and territories joined the national hospital insurance program. 


1961 Saskatchewan extended public health insurance to cover physician services outside hospitals. 
1966 The federal government introduced the Medica/ Care Act to share the cost of medical care insurance plans with 
provinces. 


1966 The federal government introduced the Canada Assistance Plan (CAP), a cost sharing plan for comprehensive welfare 
programs. The plan also covered certain health services. 


Ss 


1968-72 | Provinces and territories joined the national medical care program. 


1977 The Federal—Provincial Fiscal Arrangements and Established Programs Financing Act was enacted. Established 
Programs Financing (EPF) included transfers covering hospital insurance, medical care insurance, and post-secondary 
education; and the Extended Health Care Services Program. 


1984 Parliament enacted the Canada Health Act. 


1996 The federal government replaced EPF and CAP with the Canada Health and Social Transfer (CHST). 


1999 The Prime minister and all premiers (except Quebec’s) and territorial leaders signed the Social Union Framework 
Agreement. 
- 2000 First ministers issued a communiqué on health that committed them to clear accountability reporting to Canadians. 
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government began to flow directly to the provincial governments. This 
revenue still goes to the provinces and continues to grow as their economies 
erow. 


3.14 The Canada Health Act was passed in 1984, when Established Programs 
Financing was the main mechanism for transferring federal funds to the 
provinces for health care. The Act was a response to concerns that doctors’ 
practice of extra-billing and hospitals’ charging of user fees were creating a 
two-tiered health system that would threaten access to care. The Canada 
Health Act reaffirmed the federal government’s commitment to universal, 
accessible, comprehensive, portable, and publicly administered health 
insurance (Exhibit 3.2). 


Exhibit 3.2 Canada Health Act: Purpose and requirements 


Purpose 

The Canada Health Act aims to ensure that all residents of Canada have access to necessary health care on a prepaid basis. 

The purpose of the Canada Health Act is to establish criteria and conditions in respect of insured health services and extended 

health care services provided under provincial law that must be met before a full cash contribution may be made. 

Criteria 

1. Public administration. The health insurance plan of a province/territory must be administered and operated on a non-profit basis 
by a public authority accountable to the provincial/territorial government. 


2. Comprehensiveness. The plan must insure all medically necessary services ‘provided by hospitals and physicians and, where 
permitted, services rendered by other health care practitioners. 


3. Universality. The plan must entitle 100 percent of eligible residents to insured health services on uniform terms and conditions. 


Portability. Residents are entitled to coverage when they move to another province/territory and when they travel within Canada 
or abroad (with some restrictions). 


5. Accessibility. The plan must provide reasonable access to insured hospital and physician services on uniform terms and 
conditions. Additional charges to insured patients for insured services are not allowed. No one may be discriminated against on 
the basis of income, age, health status, etc. 5 


Conditions 
1. Provision of information. Provincial/territorial governments are required by regulations to provide annual estimates and 
statements on extra-billing and user charges. They are also required to voluntarily provide an annual statement describing the 


operation of their plans as they relate to the criteria and conditions of the Act. This information serves as a basis for the Canada 
Health Act Annual Report. 


2. Provincial recognition of federal contributions. Provincial/territorial governments are required to give public recognition of 
federal transfers. 

Provisions on extra-billing and user charges 

1. Extra-billing for an amount in addition to any amount paid or to be paid for an insured health service by the health care 
insurance plan of a province. 

2. User charge for an insured health service that is authorized or permitted by a provincial health care insurance plan that is not 
payable, directly or indirectly, by the plan, but does not include any charge imposed by extra-billing. 

Penalty provisions 


1. Mandatory financial penalty for extra-billing and user charges. Direct patient charges are subject to dollar-for-dollar deductions 
from federal transfer payments. 


2. Discretionary financial penalty for non-compliance with the five criteria and two conditions. Financial penalties will reflect the 
gravity of the default. 


Source: Health Canada, Canada Health Act,Annual Report, 1997-98 
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3.15 In 1996, the Canada Health and Social Transfer (CHST) was 
introduced to replace Established Programs Financing and the Canada 
Assistance Plan. The CHST is a block fund or lump sum transfer from the 
federal government to the provinces and territories on a per capita basis to 
subsidize health care, post-secondary education, and social assistance. The 
provinces allocate the block fund among these social programs according to 
their own priorities. 


The federal government influences health care delivery through the Canada Health Act 
and the Canada Health and Social Transfer 


3.16 The federal government does not legislate health care directly. Instead, 
through the Canada Health Act, it supports the conditional transfer of 
payments to the provinces and territories for health care. The Canada Health 
and Social Transfer, administered by the Department of Finance through the 
Federal-Provincial Fiscal Arrangements Act, is a means by which the federal 
government influences health care in Canada. 


3.17. By imposing conditions on the transfer of funds through the CHST, the 
Canada Health Act seeks to ensure that medically necessary services will be 
universally accessible, without direct charge, to residents of all Canadian 
provinces and territories. The conditions the Act imposes are its five criteria 
(Exhibit 3.2). The Act states that, in order that a province may qualify for a 
full cash contribution... [under the CHST], the health care insurance plan of 
the province must satisfy the criteria respecting: (a) public administration; 
(b) comprehensiveness; (c) universality; (d) portability; and (e) accessibility. 


3.18 The five criteria of the Canada Health Act reflect national objectives. 
Health Canada’s role is to assess the extent to which health care delivery in 
the provinces and territories complies with the Act’s criteria and provisions 
and to authorize the payment of the CHST based on that assessment. 


There are variations in the delivery of publicly funded health care services 


3.19 The Canada Health Act covers hospital, physician, and surgical-dental 
services in a hospital that are judged to be medically necessary and requires, 
under the criterion of comprehensiveness, that these services be insured by 
provincial health care insurance plans. Achieving comprehensiveness does 
not ensure the public funding of the same set of health care services in every 
province. There are a number of reasons for a lack of uniformity. First, the 
term “medically necessary” is not defined in the legislation, and may be 
interpreted differently in each province. Further, the provinces and territories 
do not use a uniform method for determining which services are medically 
necessary. Similarly, each province and territory is separately and 
independently responsible for its own decisions to delist any medical services, 
that is, to no longer pay for them through the public health care insurance 
plan. Finally, each provincial health care insurance plan covers additional 
services that are not covered by the CHA but are publicly funded through the 
provincial plan. These additional services vary from province to province. 


3.20. For all of these reasons, there are variations across the country in 
public coverage of certain health care services. For instance, the removal of 
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varicose veins and eye examinations for people aged 19 to 64 are medical 
services that are insured in some provinces but not in others. The extent and 
the impact of variations across the country are not known. 


Recent increases in public health care expenditures 


3.21 


During the period 1991 to 1997, the proportion of public to total 


expenditure on health care declined. After 1997 the trend reversed, because 
the federal government announced significant increases in the CHST 


(Exhibit 3.3). 


3.22 


The pattern of health care spending has changed over the past decade. 


Spending on hospital and physician services has become a smaller percentage 
of total health care spending. These services are covered by the Canada 

Health Act. Health care expenditures for drugs have increased as a percentage 
of total health care expenditures. Drugs are not covered by the Canada Health 


Act unless they are administered in a hospital. 


3.23 


In 1984, when the Canada Health Act was passed, hospitals provided 


many services such as drugs, rehabilitation, convalescent care, and palliative 


care that today are delivered increasingly in the home or community. And 


they are delivered by a broader range of health care providers, such as nurses, 


nurse practitioners, physiotherapists, and occupational therapists. These 
services fall outside the scope of the Canada Health Act, and their delivery is 
often paid for by patients or by private health insurance plans. 


Exhibit 3.3 Health care spending in Canada, 1991 to 2001 


2001 
1991 1993 1995 1997 1999 (Forecast) 
Total health care expenditure ($ billions) 66.2 HONS 74.1 ey) 89.5 LO275 
Public health care expenditure ($ billions) 49.35 51k95 Sy2 {5} 55 63.4 74.5 
Private health care expenditure ($ billions) 16.89 19.56 23 239 26.1 28 
SS a Ea 

Total health care expenditure as a percentage 9.6 OS C1 il 8.9 QP 9.4 
of GDP 
Public health care expenditure as a percentage 74.5 Wien TAGS WO 70.8 YEH 
of total 
Private health care expenditure as a percentage PASS) 273 28.7 2S 2002 TRS, 
of total 
Total health care expenditures by use of funds 
as a percentage of total 

Hospitals 38.7 37.4 34.5 33:0 31.9 S1n5 

Physicians 15.4 14.7 14.3 14.2 13.6 1325 

Drugs (prescribed and non-prescribed) 11.6 WA 1325 14.4 14.9 PoeZ 

Other 34.3 35.2 Sil: 38.4 39.6 39.8 

= ol 
Source: Canadian Institute for Health Information 
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Weaknesses we identified in 1999 
3.24 Our 1999 audit examined the Canada Health Act and the Canada 


Health and Social Transfer, the federal government’s key mechanisms of 
support to the provinces and territories in their delivery of health care. Our 
1999 Report (Chapter 29) discussed the way the government had used these 
tools as instruments of public policy, the purposes they had served, and their 
effectiveness in achieving the results they were designed to achieve. 


3.25 We noted several weaknesses in Health Canada’s reporting, 
monitoring, and enforcement activities, such as the following: 


« The federal contribution to health care was not being reported to either 
Parliament or the Canadian public. 


* Health Canada did not have the information it needed for effective 
monitoring of provincial and territorial compliance with the Canada 


Health Act. 


¢ The Department was not reporting the extent to which the provinces 
and territories were complying with the Act. 


¢ It was not rigorously enforcing the Act. 


Focus of the follow-up 


3.26 Our follow-up audit reviewed the current administration of the Canada 
Health Act in light of our 1999 observations and recommendations. 


3.27. We looked specifically at Health Canada’s activities and systems for 
collecting information. We assessed the Department’s capacity to collect and 
use information to monitor and report on compliance with the Act. We 
examined the enforcement of the Act. Finally, we examined the Department’s 
development of a mechanism for resolving disputes with a province or 
territory. Our report presents the results of the areas that were re-audited. 
Further details can be found about the follow-up’s objectives, scope, 
approach, and criteria in About the Follow-Up at the end of the chapter. 


Observations and Recommendations 


Federal contribution to health care = 3.28 =~ In our 1999 report, we observed that the federal government could not 
is still unclear say how much it contributed, in total, to health care. We recognized that the 
design of the CHST as a block transfer meant that Parliament and the general 
public also had no clear idea of the amount of federal funding that was 
directed to health care. We also made clear our expectation that the federal 
government be in a position to provide Canadians with information on its 
contribution to health care. 


3.29. The design of the CHST as a block transfer reflects a federal policy 
decision to give the provinces and territories more flexibility to allocate 
resources among the designated areas of social funding. Although the total 
amount allocated to social programs is known, Canadians do not know what 
portion of the CHST the federal government intends to contribute to health 
care. 
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3.30 In 2000, the Department of Finance made a number of estimates of the 
federal government’s support to health provided through transfers to the 
provinces and territories. One of the estimates assumed the same historical 
allocation to health that prevailed under the transfer system prior to 1996-97 
(43 percent cash and 68 percent tax points, for a weighted average of 

54.3 percent). The other estimates suggested that the allocation might be 
higher. The Department of Finance told us that the estimate based on 
historical allocation was made for illustrative purposes only. In recent years, 
the federal government has invested additional funds, primarily for health, 
through the CHST (Exhibit 3.4). In July 2002, the Department of Finance 
made a new estimate based on provincial spending on health and other social 
programs covered by the CHST: Using the same breakdown, it estimated that 
62 percent of the CHST ($21 billion in 2001-02), on average, is spent on 
health annually. 


3.31 No distinction is made in the CHST to indicate how much is intended 
for each of the social programs it funds. The federal transfer is a combination 
of cash contributions and tax points for the delivery of health care, post- 
secondary education, and social assistance (Exhibit 3.5). There is no agreed- 
upon estimate that captures the federal contribution to health care. The 
Canadian public has not had a clear idea of the amount of federal funding 
directed to health care. Nor can the federal government say what its total 
contribution to health care will be. Consequently, parliamentarians must 
make decisions about federal support of health care delivery without 
adequate information on the federal contribution. 


Exhibit 3.4 Recent federal investments in health care 


1999 | The federal Budget announced increased funding for the Canada Health 
z and Social Transfer (CHST) of $11.5 billion over five years, specifically for 
| health care. 


= “2000 -| The Budget announced a $2.5 billion increase in the CHST over four years 
a : : to help provinces and territories fund post-secondary education and health 
| care. 


| First Ministers agreed on an action plan for renewing health care and 
investing in early childhood development. The federal government 

| committed to invest an additional $21.1 billion in the CHST over five 
: years, including $2.2 billion for early childhood development. It would 
| also invest in three targeted areas: 


¢ $1 billion in 2000-01 and 2001-02, in transfers to the provinces and 
territories for new medical equipment; 


* $800 million over four years, beginning in 2001-02, in a renewed 
Health Transition Fund to support innovation and reform in primary 
care; and 


= | _ * $500 million to establish an independent corporation mandated to 
ee en accelerate the development and adoption of modern systems of 
information technology, such as electronic patients’ records. 


Source: Department of Finance 
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Exhibit 3.5 Canada Health and Social Transfer (CHST) 


($ billions) 
EPF/CAP* CHST 


31.9 
30.0 


26.8 
25.8 ; 


12.5 12.5 14.5 15.5 


10.7 114 12.2 13.3 143 15.6 16.4 16.1 16.6 17.2 17.9 18.8 


1994-95 1995-96 1996-97 1997-98 1998-99 1999-2000 2000-01 2001-02 2002-03 2003-04 2004-05 2005-06 


|| Cash increase Fa Base cash &B Tax points 


*EPF Established Programs Financing 
CAP Canada Assistance Plan 


Source: Department of Finance, at 22 July 2002 


3.32. Public opinion polls show that Canadians value Medicare and are 
concerned about its long-term sustainability. There is considerable debate 
over the federal contribution to health care. Much of the debate has been 
about incremental funding—that is, reductions in or additions to the federal 
contribution. 


3.33 In 2000, the federal government put an additional $21.1 billion into 
the CHST for five years. This increase was targeted primarily to support 
health care delivery in the provinces and territories. When the increase was 
announced, First Ministers agreed to improve public reporting in their own 
provinces as a way to inform Canadians about some of the effects of health 
care spending. 


3.34. The current public debate on the future of health care is evidence of 
Canadians’ concern about their health care system. The debate is limited by a 
lack of sufficient information on the federal contribution to health care 
funding: Canadians do not know how much the federal government 
contributes. We would expect the federal government to be in a position to 
tell Canadians what it intends to contribute to health care funding in the 
future. This information could help to inform the current debate on health 
care reform. Further, federal efforts to spell out its own intent would not affect 
the flexibility of the provinces and territories to allocate the CHST block 
funding according to their own priorities. 
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3.35 | Recommendation. The federal government should provide sufficient 
information to Parliament to allow for informed debate on future health care 
funding. 


Government’s response. The federal government provides full information 
on its own direct health spending through the Main Estimates and the Health 
Canada Web site. It also provides full information (in Budget booklets, the 
Finance Canada Web site, and the Main Estimates) on its transfer payments 
to provinces and territories, which, in the case of the Canada Health and 
Social Transfer (CHST), is provided to support provincial and territorial 
spending on health care, post-secondary education, and social assistance and 
social services, including early childhood development. The policy intent in 
establishing the CHST block fund is to provide provinces and territories with 
the flexibility to allocate funds according to their respective priorities; in this 
context, provinces and territories have the flexibility to use all of the CHST 
cash in support of the Canada Health Act if they wish. The federal 
government has made strides in ensuring that more information is made 
available on the nature and policy objectives of transfer programs, including 
the CHST, and will continue to explore options to improve health care 
information. 


Monitoring compliance with the = 336 © We observed in 1999 that Health Canada did not routinely collect the 
Canada Health Act information it needed for determining to what extent the provinces and 
territories were complying with the Canada Health Act. We noted that its 
sources of information were not sufficient to determine the extent of 
compliance. We also noted that the Department had not collected reliable 
information on relevant indicators. 


3.37 We expected Health Canada to use a monitoring system that collected 
sufficient relevant information on compliance with the Canada Health Act, 
analyzed it, reported it, and provided the Department with the information it 
needed to administer the Act. 


Increased monitoring capacity 


3.38 A reorganization at Health Canada in 2000 created the Canada Health 
Act Division. The new division’s major responsibilities include monitoring 
provincial and territorial health insurance plans to ensure that they meet the 
criteria and conditions of the Act; informing the Minister of possible non- 
compliance and recommending appropriate action to resolve it; providing 
information to senior Health Canada officials on the monitoring and 
interpretation of the Act; reporting annually to Parliament on provincial and 
territorial compliance with the Act; and administering and enforcing the Act. 


3.39 Inthe Canada Health Act Division, annual funding was increased from 
$2.5 million in 1999 to $4 million; the additional funding was to support 
activities in the Department’s regions and at headquarters. Two new positions 
were created in each of Health Canada’s six regions, and the total staff 
complement of the Division rose from 23 to 49. We were told that the new 
positions are responsible primarily to collect information for the purposes of 
monitoring compliance with the Act. 
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3.40 At headquarters, Health Canada established the Information, Analysis 
and Reporting Unit, responsible for collecting and analyzing information to 
improve the Department’s capacity to monitor provincial and territorial 
compliance with the Canada Health Act. The Unit developed the Canada 
Health Act Information System to facilitate the storage, tracking, and 
analysis of information related to compliance. The unit is also responsible for 
producing the Canada Health Act Annual Report. 


Limited number of information sources for monitoring 


3.41. Our follow-up found that the sources of information Health Canada 
uses for monitoring purposes have not changed substantially since our 1999 
audit. There are three major sources of information available to the regional 
analysts—third-party sources, public complaints, and voluntary submissions 
by the provinces and territories for the purpose of the annual report. 


3.42 Third-party sources include the media, stakeholders’ publications, 
personal and professional contacts, and conferences. The Department’s 
regional analysts rely heavily on these sources to track and identify potential 
cases of non-compliance in the provincial delivery of health care. These 
sources still do not allow the Department to assess the extent of provinces’ 
and territories’ compliance with the Canada Health Act. 


3.43 Complaints by individuals are the second major source of information 
on potential cases of non-compliance. Complaints trigger queries and 
investigations that can lead to the identification of cases of non-compliance 
with the conditions of the Canada Health Act. Complaint-driven monitoring 
assumes that where no complaints are reported, conditions are being met. 


3.44 Finally, the provinces and territories submit information voluntarily to 
Health Canada each year, according to guidelines developed by the 
Department. This information describes the operation of the provincial and 
territorial health care insurance plans in relation to the Act. The information 
is analyzed by the Department to ensure that provincial and territorial health 
care insurance legislation comply with the Canada Health Act. 


3.45 As in 1999, we found that these sources alone do not provide sufficient 
information to monitor compliance and determine the extent of compliance. 
Information the Department provided to us shows that none of the 
investigations of potential non-compliance initiated since 1999 has been 
related to the criteria of the Act. All new investigations reported to us have 
dealt with the provisions of the Act, that is, user charges and extra-billing. 
The fact that there are no investigations related to the criteria of the Act 
raises some questions. 


3.46 We are concerned that there may be cases of non-compliance with the 
criteria of the Act that Health Canada has not investigated. 


Limited access to the information that could identify non-compliance 


3.47. There are no regulations that require the provinces and territories to 
submit specific information to Health Canada, apart from extra-billing and 
user charges. Health Canada attempts to monitor health care delivery in the 
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provinces and territories in the absence of required reporting. While the Act 
places conditions on the transfer of CHST funds to the provinces for health 
care delivery, the legislation does not include regulations to facilitate 
monitoring for provincial compliance with these conditions. When the Act 
was passed in 1984, regulations were drafted to require reporting; however, 
they were never promulgated. As a result, the Department has only limited 
ability to obtain the information it needs from the provinces. 


3.48. The Canada Health Act defines the conditions for the federal transfer 
of funds to the provinces for health care. It also includes a provision for 
obtaining the information Health Canada needs from the provinces and 
territories. That provision has not been exercised. In its interactions with the 
provinces and territories, Health Canada has tended to take a non-intrusive 
approach to administering the Canada Health Act. 


3.49 The Department's regional analysts informed us that they cannot 
collect information from medical facilities because delivery of services is 
under provincial and territorial jurisdiction. The provinces and territories are 
responsible for ensuring that health care delivery complies with their own 
health care legislation; they are not obliged to report this information to 
Health Canada. Thus, Health Canada does not have access to the 
information it needs to administer the Act. 


3.50 We observed that other organizations collect data on health care 
delivery. The Canadian Institute for Health Information (CIHI) has collected 
information on waiting times in hospitals; Statistics Canada conducts the 
Canadian Community Health survey; and organizations outside government, 
such as the Canadian Medical Association and the Canadian Healthcare 
Association, also collect health statistics. While these organizations do not 
have the same mandate as Health Canada, they do collect some information 
that Health Canada could use for monitoring but currently is not. 


Health Canada still does not report the extent of compliance 


3.51 We noted in 1999 that previous editions of the Canada Health Act 
Annual Report to Parliament had not met the Act’s requirement that Health 
Canada indicate to what extent each province and territory had satisfied the 
criteria and conditions in the Act. The annual reports were descriptive and 
narrative, and they did not provide enough information for Parliament to 
determine the extent of provincial and territorial compliance. 


3.52 The Minister of Health has an obligation under the Act to report on its 
administration and operation, including all relevant information on the 
extent to which the provinces have satisfied the conditions for payment 
under the Act. 


3.53 The Canada Health Act Annual Report contains general descriptions 
of the non-compliance issues under investigation. It also shows, by province 

and territory, penalties levied during the previous year and annual deductions 
since the passage of the Canada Health Act. 
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3.54 Neither of the two most recent annual reports (1999-2000, 2000-01) 
contains an assessment of the extent to which the provincial health care 
insurance plans comply with the Canada Health Act. The provinces and 
territories voluntarily submit information to Health Canada as prescribed by a 
template developed by the Canada Health Act Division. This information is 
reported in the Canada Health Act Annual Report but is not used to 
determine compliance. The reports provide good descriptions of provincial 
and territorial health care insurance legislation. Health Canada does not 
conclude to what extent this legislation complies with the Canada Health Act. 
The reports also provide statistics on health care delivery. Health Canada 
does not conclude to what extent provincial and territorial health care 
delivery complies with the federal legislation. As a result, members of 
Parliament cannot determine from the Canada Health Act Annual Report 
whether the spending of billions of dollars transferred to the provinces and 
territories results in health care delivery that meets the intent of the Act. 


First Ministers agree to report publicly on performance of their health care systems 


3.55 In September 2000, First Ministers issued a communiqué on health 
that committed them to clear accountability reporting to Canadians. This 
accompanied the announcement of $21.1 billion in new federal investments 
over five years to support health care renewal. 


3.56 First Ministers directed their health ministers to collaborate on 
developing 14 indicator areas against which each government would begin 
reporting by September 2002. These indicator areas are to address health 
status, health outcomes, and quality of service. Public reporting would 
include verification by an independent party. 


3.57. Each jurisdiction (federal, provincial, and territorial) was to report to 
its constituents in September 2002 on the 14 indicator areas agreed on. This 
initiative represents a step in the right direction. Among the indicators, two 
relate to compliance (hospital waiting times and access to first-contact health 
services). If the governments pursue this initiative and continue to 
collaborate on developing additional indicators, this type of reporting could 
provide Canadians with information on the public administration, 
universality, accessibility, comprehensiveness, and portability of health care. 


3.58 Health Canada recognizes that there are shortcomings in the 
collection of health information. First Ministers are committed to working 
together on the development of a comprehensive framework using jointly 
agreed comparable indicators to address health status, health outcomes, and 
quality of service. To improve monitoring and reporting requires the 
collaboration of provinces and territories with the federal government. 


3.59 Recommendation. Health Canada should, in collaboration with the 
provinces and territories, fulfil its obligation to administer the Canada Health 
Act by collecting the information it needs to enforce and report compliance 
with the Act. 
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Health Canada’s response. Health Canada is already collecting considerable 
information on the activities of provincial and territorial health insurance 
plans. This information is used to assess provincial and territorial compliance 
with the criteria and conditions of the Canada Health Act and to report to 
Parliament. 


Any gaps with regard to information collection will be addressed in the 
context of the provisions of the Canada Health Act Dispute Avoidance and 
Resolution process agreed to by the provinces and territories in April 2002. 
The process allows for governments to engage in discussions, information 
exchange, and joint fact-finding to ensure that they have all the necessary 
information to assess concerns with regard to the interpretation of and 
compliance with the Canada Health Act. 


When non-compliance has been identified, resolution continues to be slow 


3.60 In 1999 we reported that the resolution of non-compliance issues was a 
slow process. We cited examples of suspected non-compliance that had 
remained unresolved for a number of years. At the time of that report, there 
were 14 unresolved issues. Three of them have since been resolved; the 

11 remaining issues have been outstanding for five years or longer. 


3.61 The process for resolving federal/provincial differences over 
compliance with the CHA has remained slow. Since 1999, Health Canada 
has identified 12 more cases of suspected or confirmed non-compliance and 
has attempted to deal with them through means other than penalties. Two of 
the issues have been resolved; 10 remain outstanding. 


3.62 We have been told that the slow resolution of disputes is partly due to 
difficulties encountered in the interpretation of the Canada Health Act. We 
note that Health Canada has developed the Canada Health Act Policy 
Interpretation Manual for employees of the Canada Health Act Division. 
However, the Department has not shared the manual with the provinces and 
territories because of legal constraints. Consequently, different interpretations 
of the Act at various levels of government have hindered the timely 
resolution of problems. Enforcement of the Canada Health Act, whether 
through penalties or negotiations, could be facilitated by an attempt to ensure 
timely and ongoing discussion of the Act’s interpretation by all parties. 


3.63 In 1999 we recognized the need for federal/provincial/territorial 
collaboration to avoid disputes or resolve them, as a means of enforcing the 
Act. Co-operation and collaboration were the major themes of the Social 
Union Framework Agreement signed in February 1999 by the Prime Minister 
and all premiers (except Quebec’s). The Agreement describes how federal, 
provincial, and territorial governments will work together to sustain Canada’s 


social programs. 


3.64 In the spirit of the Agreement, Health Canada is committed to 
administering the Canada Health Act in a non-intrusive manner and in 
collaboration with provincial and territorial governments. To this end, the 
Department has worked with the provinces and territories to develop a 
process to facilitate the avoidance of disputes and the resolution of non- 
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compliance issues. Agreement on the process was reached in April 2002. It 
has not been used yet, but it has potential for improving collaboration to 
avoid and resolve non-compliance issues. 


When non-compliance issues cannot be resolved, penalties may be imposed 


3.65 Negotiation is the first course of action in the dispute resolution 
process. If negotiation fails, the Minister may impose a penalty. The Canada 
Health Act provides for two kinds of penalty. The first is a mandatory penalty 
related only to the provisions of the Act that cover extra-billing and user 
charges. A mandatory penalty amounts to a dollar-for-dollar deduction from 
the federal CHST payment to the province or territory that has allowed 
extra-billing or user charges. The second type of penalty is a discretionary 
penalty. Under sections 14 to 17 of the Act, discretionary penalties (CHST 
reductions) can be imposed if a province or territory has not satisfied any one 
of the Act’s criteria for health care delivery and if all reasonable efforts at 
consultation and negotiation have failed. At this point, the Minister may 
refer the matter to the Governor in Council for a decision on whether the 


CHST should be reduced. 


3.66 In 1999 we observed that enforcement of the Canada Health Act 
through penalties for non-compliance had been limited to the mandatory 
penalty for non-compliance with the extra-billing and user fee provisions of 
the Act. The federal government had never applied the discretionary 
penalties for non-compliance with the criteria of the Act. 


3.67 The Act imposes obligations on the federal government. It defines the 
conditions that the provinces must meet to receive federal payments. Ifa 
provincial health care insurance plan falls short of these conditions, the 
federal government has the legislative authority to take measures that can 
include withholding all contributions to the offending province. 


Few penalties have been imposed 


3.68 The Canada Health Act Annual Report 2001-2002 shows that the 
federal government imposed one penalty for user charges. Exhibit 3.6 shows 
the number of penalties imposed since 1995 and their total value. 


3.69 Health Canada has told us that it hesitates to impose penalties. It has 
shown a strong preference for consultation and negotiation to resolve 
disputes in collaboration with the provinces and territories. It believes that if 
it were to impose penalties, the provinces and territories could simply choose 
to absorb them and continue to contravene the Act. 


Evaluating and reporting performance 


3.70 _ In 1999 we reported that an evaluation by the Department had 
focussed on the Health Insurance Division’s procedures for monitoring 
whether provincial health care delivery met the accessibility criterion of the 
Act. The evaluation noted that it was not clear how effectively the Division 
monitored the status of the health care system, including the implications of 
emerging issues that affect the underlying principles of the Act. 
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The evaluation examined only the Division’s monitoring activities; it did not 
question, for example, whether Canadians do in fact have reasonable access 
to health services across the country. 


3.71 Health Canada agreed with our recommendation to deal with key 
issues of the Canada Health Act in future evaluations. However, it has 
suspended further evaluation of its responsibilities under the Canada Health 
Act, in light of several ongoing studies of the health care system, including the 
following: 


¢ Commission on the Future of Health Care in Canada (chaired by Roy 
Romanow), report expected in 2002 


¢ Standing Senate Committee on Social Affairs, Science and Technology: 
The Health of Canadians—The Federal Role (tabled by Senator Kirby), 
two reports in 2001 and another three to be tabled in 2002 


¢ Alberta Premier’s Advisory Council on Health (chaired by Don 
Mazankowski), report tabled January 2002 


* Saskatchewan Commission on Medicare (chaired by Ken Fyke), report 
tabled April 2001 


* Quebec’s Commission d’étude sur les services de santé et les services 


sociaux (chaired by Michel Clair), report tabled December 2000 


Commitment to reporting on performance and effectiveness 


3.72 The 1999 Budget announced the $43 million health-related Federal 
Accountability Initiative. The initiative committed Health Canada to 
becoming more accountable to Canadians for the performance of its own 
health programs. The Department lacked the capacity in 1999 to undertake 
the initiative and we recommended that it explore options to meet its 
commitment. 


3.73 Our follow-up found that Health Canada has done some work on this 
initiative, for example, its Performance Measurement Development Project. 
However, it has not completed the project; it has not yet fully developed 
performance expectations and reported against them. The Department was 
not able to provide us with an action plan and target dates. 


3.74 Recommendation. Health Canada should ensure that the Federal 
Accountability Initiative provides the required performance measures of its 
own programs and reports against them. 


Health Canada’s response. Health Canada’s investment in the Federal 
Accountability Initiative has served to advance the Department's policies and 
practices with respect to performance measurement and reporting to 
Canadians. 


Health Canada is meeting its commitments to develop indicators and report 
on them. As part of the September 2000 First Ministers’ commitment to 
regular federal/provincial/territorial reporting on health system performance, 
Health Canada is preparing a report on performance, including data on First 
Nations health as well as information with regard to veterans, inmates, and 
military personnel. This report will be published in September 2002. 
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Health Canada is working expeditiously on the development of performance 
indicators for the Department, which will be the basis for further 
improvements in performance reporting. The indicators will be proposed for 
internal approval in September 2002 to be used in the departmental 
performance report in the following year. 


Conclusion 


3.75 Weaknesses in the information that Health Canada collects and 
reports annually to Parliament on the administration and operation of the 
Canada Health Act remain a long-standing problem. Increased monitoring has 
not been able to remedy the problem of limited sources of information. 
Health Canada does not use the information it collects to assess or report on 
the extent of provincial and territorial compliance with the criteria and 
conditions of the Canada Health Act. 


3.76 Health Canada has tended to take a non-intrusive approach to 
administering the Act. However, this approach has not brought about the 
speedy resolution of issues related to non-compliance with and interpretation 
of the Act. The majority of the non-compliance issues identified by Health 
Canada over the past 10 years have remained unresolved for five years or 
longer. Few penalties have been levied for non-compliance with the 
provisions of the Act. No penalties have been levied for non-compliance with 
the criteria of the Act. 


3.77 Health Canada has suspended its evaluation of programs in this area 
because of the number of ongoing studies of the health care delivery system. 
The Department remains committed to measuring and reporting on 
performance. However, it has not yet completed the Performance 
Management Development Project. 


3.78 The federal government needs to provide Parliament with adequate 
information on its contribution to provinces and territories for health care. 
Meaningful participation in the current debate on health care depends on a 
better appreciation of the federal contribution to this social program, one that 
Canadians have said helps to define this country. 
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About the Follow-Up 


Objective 
The objective of this follow-up audit was to identify Health Canada’s progress in determining the extent to which 
federal efforts to support and monitor health care delivery 

* reflect clear objectives, 

* adequately report performance, and 


¢ facilitate review and evaluation. 


Scope and approach 


The follow-up audit focussed on the recommendations made in our 1999 Report, Chapter 29, Federal Support of 
Health Care Delivery. Health Canada has undertaken a number of new initiatives in the last three years, which we 
also reviewed. 


We reviewed a status report by the Department on the action it has taken in response to our recommendations. We 
carried out extensive interviews with Health Canada staff involved in the administration of the Canada Health Act 

and with staff of the Department of Finance and the Privy Council Office. We also met with stakeholders and Health 
Canada staff in some regions. Finally, we reviewed documentation related to reported activities. 


Our report presents the results of the areas that were re-audited. 


Criteria 
We expected that Health Canada would have made satisfactory progress in implementing our recommendations. 


The criteria from the 1999 audit remain relevant. Therefore, we expected to find the following: 


* acommitment to achieving consensus among the federal, provincial, and territorial governments on objectives, 
roles and responsibilities, and standards; 


* a process for collecting and reporting information on performance; and 

* aprocess for interpretation and enforcement of the Canada Health Act that is effective and transparent. 
Ratings 
We assessed the action of The Departments or Agency against our original audit recommendations (see Key Message 
at the beginning of the Chapter). We used the following ratings: 

* Completed. Corrective action has been fully implemented 

* Satisfactory progress. Progress is being made at a satisfactory pace 

* Limited progress. Some progress is being made, but the pace or scope is not satisfactory 


* No progress. No evidence of progress although the department or agency accepted the recommendation from 
the original audit 


* Rejected. The department or agency did not accept the recommendation from the original audit 


* Unknown. Status of progress is unknown or information is not available 
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Audit team 


Assistant Auditor General: Maria Barrados 
Principal: Patricia MacDonald 
Director: Linda Anglin 


Theresa Bach 
Jacques Maziade 


For information, please contact Communications at (613) 995-3708 or 1-888-761-5953 (toll-free). 
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National Defence 
NATO Flying Training in Canada 


The audit work reported in this chapter was conducted in accordance with the legislative mandate, policies, and practices of the 
Office of the Auditor General of Canada. These policies and practices embrace the standards recommended by the Canadian 
Institute of Chartered Accountants. 


Table of Contents 


Key Message 


Introduction 


Focus of the follow-up 


Observations 


The NATO Flying Training in Canada program 


International participation is essential 


National Defence states that the contractor is not providing the required levels of service 
in basic flying training Phase || 


The program is not providing enough training flights per day to meet requirements 
National Defence needed more instructors than originally planned 


Problems and delays in the basic flying phase of the training program have caused 
shortfalls and cancellations in subsequent training phases 


More pilots were awaiting training and were waiting longer 


National Defence and PWGSC have been working on fixing problems for over two years 
but cannot say when problems will be resolved 


National Defence has paid about $65 million for unused training capacity 
About 79 percent of the contract fees are paid regardless of how many pilots are trained 
Firm fixed payments are unconditional and irrevocable 


National Defence has not yet determined whether it has under or over paid the variable fees 
to the prime contractor 


Payments are not tied to performance 
There are no mechanisms in the contract to allow for changes 


The NATO Flying Training in Canada program did not include management practices 
consistent with a program of this size and risk 


The program is slow in devolving a management framework to the users 
National Defence is looking into new management arrangements for the program and lessons learned 


Procurement Reform initiatives are underway but progressing slowly 


Conclusion and Recommendations 
About the Follow-Up 


Appendices 


A. Excerpts from our 1999 Report, Chapter 27, National Defence—Alternative Service Delivery 
B. National Defence’s pilot training process 
C. NATO Flying Training in Canada program milestones 
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National Defence 


NATO Flying Training in Canada 


Assistant Auditor General: Hugh McRoberts 
Principal: Wendy Loschiuk 


Key Message 


41 National Defence has entered into a $2.8-billion contract to train pilots over 20 years as part of the NATO 
Flying Training in Canada program. During the first two years of program implementation, National Defence used 
only about 41 percent of the training capacity that it paid for. Because of the restrictive terms of the contract and the 
problems encountered during start up, about $65 million in training costs were paid for training that was not 
obtained. The Department has informed us that there are contractual opportunities to recoup some of this expense. 


These opportunities need to be pursued now. 


ORIGINAL ISSUES PROGRESS RATING* 


4.2 The Treasury Board Secretariat should develop 
guidelines and training for large service contracts for 
multi-year terms as part of its continuing work on 
procurement reform. 


In December 2001, we found that work on procurement 
reform had not addressed our concerns about partnering. 
The Treasury Board Secretariat’s new policy on 
alternative service delivery provides some helpful 
guidelines, but is general in nature. Guidance on 
strategic direction and best practices in complex 
procurement has been completed and according to the 
Treasury Board Secretariat complementary training will 
be provided. 


LIMITED 
PROGRESS 


4.3 National Defence did not train the contracted number of student pilots during the first two years of the NATO Flying Training in 
Canada program. In total, National Defence only used 41 percent of the training slots it paid for from February 2000 to 

December 2001. In spite of this, National Defence continued to pay its full fees for those years due to the nature of the contract. 
The total amount paid was $179.5 million for the right to use 355 training slots but National Defence used only 136 and sold off 10. 


4.4 Weare concerned that similar problems can occur in the future as the Department enters into other long-term support service 
contracts. In 1999 we reported to Parliament on two contracts for training that also showed the types of problems found in this 
follow-up. 


*Possible ratings are completed, satisfactory progress, limited progress, no progress, rejected, unknown. (See About the Follow-Up for an explanation of 
the ratings.) 


The Department has responded. National Defence agrees with most of our conclusions but feels that because of 
the unique nature of this program, many of the issues we raise were start-up challenges that were to be expected. Its 
responses, presented at the end of the chapter, elaborate on the actions it will take to address our concerns and 


recommendations. 
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NATIONAL DEFENCE—NATO FLYING TRAINING IN CANADA 


Introduction 


45 This follow-up on the NATO Flying Training in Canada program is an 
audit of National Defence’s implementation and management of the 
$2.8-billion contract for a 20-year term for military pilot training. Our 1999 
audit reviewed the contracting process up to the signing of the contract in 
May 1998. Pilot training was planned to start in February 2000. This follow- 
up reports on the results achieved during the first two years after planned 
start-up. 


46 In 1999, we reported that the contract was awarded without 
competition, that the profit mark-up was not consistent with current 
Government of Canada guidelines or supported by adequate analysis of 
contractor risk, and that the chosen financial arrangements increases some 
risks (Appendix A). Departmental officials responded that a major advantage 
of moving to a training system supported by a contractor was the shifting of 
risk to the contractor. However, they had not quantified this risk before 
awarding the contract. 


47 In our December 2001 follow-up report on National Defence— 
Alternative Service Delivery we reported that National Defence had changed 
its approach to alternative service delivery projects and that employee training 
was now available. We mentioned that National Defence did not address 
alternative service delivery projects in its Report on Plans and Priorities and 
that the government’s work on procurement reform had failed to address our 
original concerns about large service contracts for multi-year terms. 


48 This follow-up focusses on the NATO Flying Training in Canada 
program. We assessed whether National Defence had used the levels of 
service paid for from February 2000 to December 2001 and whether it had 
received value for money. As well, we followed up on our 1999 
recommendations to the Treasury Board Secretariat and Public Works and 
Government Services Canada’s continuing work on procurement reform. 


Focus of the follow-up 


49 The objectives of this follow-up were to report to Parliament on the 
implementation and management of the NATO Flying Training in Canada 
contract from February 2000 to December 2001. During the contracting 
process, the Department determined how many pilots would be trained by the 
system for the next 20 years. We did not audit the planning process or the 
numbers that were agreed to by the Department. Rather, our audit examined 
whether National Defence used the services that it paid for and whether it 
obtained value for money. 


410 We did not audit the activities of the prime contractor. Our audit was 
limited to National Defence, Public Works and Government Services 
Canada, and the Treasury Board Secretariat. We reported facts regarding 
contract delivery, but limited our comments and observations to whether the 
actions of the government were appropriate. Further details can be found in 
About the Follow-Up at the end of the chapter. 
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The NATO Flying Training in Canada 
program 


CT-156 Harvard II aircraft used for Phase || 
basic flying training. 


CT-155 Hawk aircraft used for Phase III 
advanced and Phase IV fighter lead-in 
training. 


Observations 


411 The program provides flying training to military student pilots to fly the 
Canadian Forces’ helicopters, multi-engine aircraft and the CF-18 Hornet jet 
fighters. The program trains pilots who have already passed primary flying 
training by offering four additional phases of training—basic Phase IIA, basic 
Phase IIB, advanced Phase III, and fighter lead-in training Phase IV 
(Appendix B). The first three training phases are taught at Moose Jaw, 
Saskatchewan and fighter lead-in training is taught at Cold Lake, Alberta. 
Those who will fly the CF—-18 Hornet take all four phases of the program. The 
pilots who will fly helicopters and multi-engine aircraft go on to a separate 
training program in Portage la Prairie after they graduate from basic flying 
training Phase IITA. 


412 The Harvard I] aircraft is used for basic flying training. The Hawk 
aircraft is used for advanced Phase III and fighter lead-in training Phase IV. 


413 As the prime contractor, Bombardier Inc. co-ordinated start-up 
activities with several international firms. The prime contractor is responsible 
for providing and supporting aircraft and simulators. They also provide 
ground school training for phases IIA, IIB, and III in Moose Jaw, classroom 
training systems, and maintenance services. The prime contractor provides 
the aircraft by leasing them for the life of the program from a not-for-profit 
company created at the start of the NATO Flying Training in Canada 
program. 


414 National Defence and the prime contractor share some responsibilities: 
design of ground school training, scheduling of student facilities and aircraft, 
and the provision of operational and base support services. 


415 National Defence is responsible for overall program management 
including operational control of training, providing instructors for flight 
training, providing infrastructure and military flying training areas at Moose 
Jaw and Cold Lake, and for the design, development, and conduct of flying 
training. It is also responsible for ground school training for the fighter lead-in 
phase, quality assurance, and student administration. 


416 National Defence selected this as an alternative to training its pilots on 
the aging Tutor aircraft. It was one of three options for training pilots that the 
Department studied in the early 1990s (Appendix C). The other two options 
were to either purchase fighter pilot training off-shore or to upgrade the 
Tutors and extend the original program. 


International participation is essential 


4.17 Government approval of the program depended on the participation of 
other nations to help offset the costs of new aircraft and simulators. Without 
international participants, National Defence would be responsible for all the 
costs of the program. The government directed that the established fees 
charged for foreign participants would be on a full cost recoverable basis. As 
of December 2001, Canada and the air forces of four other countries were 
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CT-114 Tutor aircraft used for pilot training 
prior to the NATO Flying Training in Canada 
program. 


NATIONAL DEFENCE—NATO FLYING TRAINING IN CANADA 


participating in the program—Denmark, the UK, Italy, and Singapore. The 


Hungarian Air Force has also signed on to the program and will start training 
pilots in 2002. 


4.18 National Defence and the participating countries pay for a specified 
number of training slots each year. For example, National Defence has 
contracted for 141.6 Phase IIA basic flying training slots each year. In 
addition, other nations have purchased training slots based on the number of 
years they have committed to the program. The levels of service the prime 
contractor is required to provide are spelled out in the various contracts and 
agreements under the program (Exhibit 4.1). 


Exhibit 4.1 NATO Flying Training in Canada program, 2001 


: 2001 Training Phase Fane Bought 
Years | Basic Basic Advanced | Fighter lead-in 

Country — committed! | PhasellA | PhasellB | Phase lll Phase IV 
Canada 20 141.6 22 32 20/22 
Denmark 20 6 5 5 5 
Italy 20 3 3 2 2 
Singapore 20oar 0) @) 6 4 
United Kingdom 10 0 6) Z 20 
Total Starts 150.6 30 47 51.2 


| Years that the country has committed to participate in the program. 


2 Canada bought 24.2 training slots and in 2001 sold 4 to other countries as part of a 
long-term sale; 20.2 slots remained. 


Source: Canada Services Agreement amendments 


National Defence states that the contractor is not providing the required levels of 
service in basic flying training Phase II 


419 During the first two years of the program (from February 2000 to 
December 2001), National Defence states that it was only able to use 105.4 
of the 265.6 Phase IIA basic flying training slots it purchased, or 40 percent. 
It was expected that during this time National Defence would have graduated 
about 216 student pilots from Phase IIA (assuming 15 percent attrition). Due 
to delays and cancellations, the expected graduation numbers were revised to 
160 Canadian pilots but by December 2001 only 61 Canadian student pilots 
graduated from this phase of the program. Many that should have finished in 
2001 will now finish in 2002. 


4.20 Prior to January 2000, the Canadian Forces trained its own pilots on 
the Tutor aircraft. Once the NATO Flying Training in Canada contract was 
signed, the Department decided to shut down its existing program prior to the 
start-up of the NATO Flying Training in Canada program. The Tutor training 
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system had produced 64 basic flying training graduates in one year (1999) 
from an intake of 78 students. By December 2001, the NATO Flying Training 
in Canada program had not reached this level of performance after two years 
of operation and when National Defence is short of pilots for its helicopters, 
multi-engine, and fighter aircraft. 


4.21. The first four courses of the NATO Flying Training in Canada program 
were originally scheduled to start between February and May 2000, but were 
cancelled because of late delivery of the aircraft and simulators. Problems in 
acquiring the Harvard II aircraft and technical data from the United States 
due to International Trade in Arms Regulations (ITAR) issues, aircraft engine 
problems, and certification problems with the aircraft and aircraft 
maintenance operation meant that the program did not start training 
students until June 2000. 


4.22 In August 2000, two months after basic flying training had started, a 
problem with the engine oil cooler caused National Defence to ground the 
Harvard II aircraft because of concerns about pilot safety. Training was 
delayed for three more months until the problem was fixed and one course 
was cancelled. 


4.23 In 2001, three more courses were cancelled. Overall 8 of the 17 courses 
planned for 2000-01 were cancelled and only 3 of the 9 courses that did take 
place were fully loaded with the contracted number of students (16). 
Furthermore, courses which should have been completed in 5 to 7 months 
took on average about 9 to 10 months to complete (Exhibit 4.2). 


Exhibit 4.2 NATO Flying Training in Canada: Basic flying training Phase IIA courses 
(February 2000—December 2001) 


Courses 
Planned | Cancelled Actual 
2000 8 5 3 
2001 9 3 6 
Total 17 8 9 


Source: National Defence 


4.24 The student intake for 2002 has been reduced to 105 students overall, 
about 41 below the contracted course-load level, in order to match students 
and available training capacity. 


The program is not providing enough training flights per day to meet requirements 


425 The contract states that on average 81 sorties, or training flights, could 
be scheduled each day by National Defence so that pilots can complete their 
flying training on schedule and move on to the next phase as planned. The 
prime contractor is responsible to provide and maintain enough aircraft and 
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simulators to meet the specified sortie rates. Departmental analysis showed 
that 17 serviceable aircraft flying five flights per day must be available to 
generate this rate of flying activity. However, on average only 14.5 serviceable 
aircraft have been available during the first two years of the program to 
maintain the sortie rate. 


426 Throughout 2001 National Defence did not receive the contracted 
sorties to meet its demand. Since the program has been in operation, 
National Defence has only scheduled an average of 62 sorties per day and has 
only flown an average of 42 sorties per day. Although there has been some 
improvement, the actual number of sorties flown per day is still far below 
anticipated levels and therefore the backlog continues to grow (Exhibit 4.3). 


Exhibit 4.3 Average sorties at the basic flying level with the Harvard II aircraft 


( Sorties scheduled 
[1 Sorties contracted 
Sorties —@— Sorties flown 


At 


January February March April May June 
2001 


ee 


Source: National Defence 


4.21 According to the Department, aircraft availability has been an ongoing 
problem and has resulted in training delays and cancellations. Departmental 
documents indicate that most of the problems with the program over the first 
two years of operation relate to Harvard II aircraft availability. The 
Department has stated that the focus is on problem-solving rather than 
attributing blame and has worked closely with the prime contractor to get the 
most training possible under current circumstances. 


National Defence needed more instructors than originally planned 


4.28 In 2001, studies conducted by the flight schools in Moose Jaw and Cold 
Lake found that the number of instructor pilots provided by National 
Defence was enough to meet the initial planned staffing levels, but this initial 
number was not adequate to meet the training demand. The flight school in 
Moose Jaw has recommended increasing the number of instructors to 80. In 
early 2002, the Canadian Forces took action to remedy the shortage by 
increasing the number of instructor pilots at Moose Jaw from 60 to 71 but a 
decision of the final level of staffing remains outstanding. 
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The CF-18 Hornet fighter aircraft. 


Problems and delays in the basic flying phase of the training program have caused 
shortfalls and cancellations in subsequent training phases 


4.29 Overall, the start-up challenges in the basic flying training phases have 
meant that National Defence has been unable to use the full capacity in the 
advanced and fighter lead-in training phases. 


430 Advanced training Phase II. The Department identified that 
advanced flying training would be underused throughout 2001 since delays in 
basic flying training meant that no students were ready to start the advanced 
work. 


4.31 By December 2001, National Defence had used only 8 of the 29 basic 
flying training Phase IIB positions it purchased and only 13 of 36 advanced 
Phase III slots. In 2001, Canada sold 5 Phase II slots that they could not use 
to the United Kingdom and Italy. On average advanced courses have been 
one month longer than planned and one course was cancelled in 2001. 


432 Fighter lead-in training Phase IV. The first course was cancelled 
because of delays in delivering the Hawk aircraft and simulator, and preparing 
the course material. However, the next three courses ran with more students 
than originally planned to make up for the cancelled course. These students 
had not completed Phase IIA or Phase IIB of this training program but were 
either previously qualified Canadian pilots being retrained to fly the CF-18 
Hornet or international students. 


4.33. One critical shortfall that was identified by National Defence in 
March 2001 was the late delivery of the centre line fuel tank for the Hawk 
aircraft, an external fuel tank which increases the range of the aircraft and is 
required for fighter lead-in training. The fuel tanks were late being built and 
once they were received, the Department needed eight months to complete 
certification test flights. This meant that the course syllabus had to be 
adjusted with shorter training flights. In some cases this lack of equipment 
extended the course by as much as two to three weeks beyond the planned 
graduation date. At the end of March 2002, the centre line tank was cleared 
for use and has now been installed on the fleet. 


4.34 By December 2001, National Defence had sold off 4 Phase IV slots to 
Denmark and Singapore, reducing its capacity to 20.2. National Defence 
used only 5 of its remaining 20.2 slots in the fighter lead-in training phase and 
had sold off 5 additional slots to the United Kingdom and Italy. The 
Canadian pilots that did graduate from Phase IV had actually received their 
basic training on the Tutor aircraft. As the program grows, by 2004 National 
Defence will have 29 Phase IV training slots, but projections to 2004 indicate 
that National Defence is only going to use 9 slots per year and continue to sell 
6 which would leave 14 slots unused. The first Canadian pilot to be trained 
from Phase IIA to Phase IV through the NATO Flying Training in Canada 
program is scheduled to graduate in July 2002. 
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More pilots were awaiting training and were waiting longer 


4.35 The number of Canadian pilots awaiting training and the wait time for 
training has increased. In September 2001, 161 students were awaiting basic 
flying training. Behind this group were another 109 candidates completing 
basic officer training, second-language training, or initial degree requirements 
whose names would be added to the pilot training waiting list. On average, 
students were waiting 18 to 22 months before starting pilot training. 


4.36 The Department has indicated that in July 2002, the number of 
Canadian pilots awaiting training had decreased to 131 and the average 
waiting time had fallen to 14 months. They expect this trend to continue. 


* 437 National Defence has stated that the operational impact has been 
manageable since the system was producing as many pilots as could be 
accepted into the helicopter and the multi-engine aircraft school or the 
various operational training units. Nevertheless, departmental documents 
show the situation was difficult for the students facing delays and frustrating 
for persons involved in the program. 


National Defence and PWGSC have been working on fixing problems for over two years 
but cannot say when problems will be resolved 


438 National Defence and Public Works and Government Services Canada 
(PWGSC) have identified a number of problems with implementing the 
contract. They are working with the prime contractor to fix them. For 
example, a steering committee, co-chaired by National Defence and the 
prime contractor was set up to address issues like the low sortie rates. 
However, many of the issues identified early in the program have yet to be 
resolved. 


439 The contract gives National Defence the right to sell some of its 
CH-146 Griffon helicopter. unused training capacity to other countries under a memorandum of 
understanding. Fees from these sales help offset some of the fixed cost to the 
Department. However, the contract limits the number of slots that can be 
sold. National Defence can only sell off 11 basic, 15 advanced and 15 fighter 
lead-in slots per year. The prime contractor nevertheless has waived this 
contractual limit during the time frame of the follow-up in order to allow 
National Defence to sell 20 slots to Hungary that cannot be used because of 
production delays in the basic flying phase. This enables the Department to 
recoup some of its costs. 


National Defence has paid about $65 million for unused training capacity 


440 National Defence has not used the full amount of training purchased 
in any phase of the program. From the start of the program to 

December 2001, National Defence only used 41 percent of the planned 
student slots but still paid most of the anticipated cost (Exhibit 4.4). 


441 National Defence has paid a total of $179.5 million, which includes 
costs for both students and instructors, of the anticipated $206.8 million 


training costs. 
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Exhibit 4.4 National Defence student use and cost of the four training phases 


Contract : 
Contracted | Actual | other | Use costs | Actualcosts | Cost 
Phase starts | (millions) | ($ millions) |  (%) 
Basic (IIA) 265.6 105.4 O 40 45.4 45.7 100 
i 
Basic (IIB) 29.0 8.0 0 28 ley ns} 100 
Advanced 36.0 13.0 5 50 Ie 14.9 95 
CD) 
+ =i 
Fighter lead- 24.2 9:0 5 58 8.3 8.1 98 
in (IV) 
| 
Total 354-60 1a 54 10 41 70.9 70.2 99 
Total costs! 206.8 179.5 87 
shee = ee eee 


‘Including costs for training instructor pilots and firm fixed fees. 


Source: National Defence, 2002 


4.42. National Defence calculated a cost per student for the foreign 
participants based on the target capacity of the program. If the program had 
been operating at its total capacity during the first two years, National 
Defence would be paying the same per student as other nations. During the 
first two years of the program, we estimate that National Defence paid about 
$65 million for training that it did not use. International participants are 
using all of the training slots they bought while Canada is not. This means 
that National Defence is paying the financial impact of inefficiencies in the 
program and the lack of productivity in Phase IT (Exhibit 4.5). 


4.43 In some cases, the minimum amount of equipment which the program 
needs has residual capacity that can train more students as the program 
expands, for example; the simulators for the Hawk aircraft. Because foreign 
fees were based on participants paying a fair share of equipment total capacity, 
until the program expands Canada is absorbing this as a business risk which 
the Department hopes to recover in the future. As well, Canada pre-pays the 
fees in advance of the training program reaching its full capacity. As of 
December 2001, the value of these two factors was estimated to be about 
$25 million (Exhibit 4.5). However, the department feels that as the program 
reaches full capacity, much of this value will be recouped. The final result will 
depend on how much the program finally expands. 


444 In our 1999 report on Alternative Service Delivery, we found that 
inflexible contract arrangements resulted in payments for unused training 
capacity at the Meaford Area Training Centre and the Canadian Aviation 
Training Centre. For example, the Meaford Area Training Centre operated 
under a $40 million, five-year, fixed price contract and had used only 

43 percent of its capacity. The Canadian Aviation Training Centre flying 
training program at Portage La Prairie, operating under a $165 million 
contract, was consistently underused during the first six years of operation. 
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Exhibit 4.5 Costs and value of unused capacity for the NATO Flying Training in Canada program 


Estimated costs based on starts and costs of the Canada Services Agreement (CSA) | —_($ millions) 
Cost of training students | Z ipa 
Cost of instructor pilots BS 
Total 1261 
Catiadian student pilots trained 43 
Canadian instructor pilots trained HS 
Slots sold to foreign countries 5 
Total 61 
Value not received due to under use of the program 65 1&2 

Actual costs of the program to Canada ($ millions) 
Actual amount that Canada has paid to the contractor 190 
Revenues from foreign countries 39 
Actual costs of the program to Canada 1513 
Less estimated cost of the program (126) 
Difference between estimated and actual cost of the program 254 


Notes: The costs reflected in this calculation do not include other operational or personnel costs incurred 
by National Defence, such as pilot salaries and overhead. 


The firm fixed fee used in estimating the CSA cost of training students and instructors and in 
calculating the value of actual output of Canadian student pilots and instructors was taken from 
foreign fee rates; the fee included an element of transition cost repayment. 


Estimated value of the Canadian training slots that were actually used and the value of slots that 
were sold to foreign countries since they could not be used by Canada. 


Actual costs for Canada and foreign revenue does not include approximately $18 million in variable 
and cost reimbursable fees, paid to the contractor, a portion of which should eventually be returned 
when the reconciliation process is complete. 


Estimated value of program overhead (for example, simulator time) that was not used in the first 
two years of operation. Canada will continue to absorb costs for this additional capacity. 


About 79 percent of the contract fees are paid regardless of how many pilots are trained 


4.45 National Defence pays tuition fees to the prime contractor for the 
services detailed in the contract. Tuition fees are based on a schedule of 
payments in the contract and are composed of fixed and variable fees. About 
79 percent of the total $2.8 billion cost of the program covers overhead and 
equipment costs and is fixed. This amount must be paid no matter how many 
pilots are trained. The remaining 21 percent of the fees are variable and are 
based on the actual number of pilots trained and their use of the aircraft and 
supplies. The contract also calls for National Defence to pay transition fees to 
the prime contractor for work done before the program was implemented 


(Exhibits 4.6 and 4.7). 
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Exhibit 4.6 Fee structure for the NATO Flying Training in Canada program 


Fixed fees 


Fixed fees cover program expenses for the delivery of training services and are payable 
whether or not National Defence uses the contracted amount of training capacity. 
There are two types of fixed fees: 


¢ Firm fixed fees. Firm fixed fees are used to obtain the assets such as aircraft, 
simulators, the provisions needed to start the program, and the initial set up costs. 
These fees are fixed for the duration of the program. 


Firm fees. Firm fees are related to the cost of running the airport at Moose Jaw, 
providing and maintaining all the program’s infrastructure, operating the ground 
school and providing the simulator instruction, providing aircraft maintenance, and 
contractor program management, and insurance. These fees are firm for the length 
of the program but are adjusted for inflation and changes in currency exchange 
rates. 


Variable fees 


Variable and cost reimbursable fees are based on actual student pilot usage and are 
supposed to be adjusted every six months. 


e Variable fees. Variable fees relate to actual use of the aircraft to cover the cost of 
consumable spare parts. A rate per flying hour has been set and this is charged 
based on the actual flying time of each student and instructor. Countries prepay an 
estimated amount which is later adjusted based on actual usage. This hourly rate 
has been set for the contract but is adjusted based on inflation. 


Cost reimbursable fees. These fees are for the actual use of petroleum and oxygen 
consumed during the program. Like the variable fees, participants prepay an 
estimated amount which is then later adjusted based on actual usage. 


Transition fees 


Transition fees cover the cost of transferring the operation of the base to the prime 
contractor and upgrading the original facilities to meet the new program requirements. 


Source: National Defence, April 2002 


Exhibit 4.7 Basis of payment over the 20-year contract period 


Amount 
Type of payment ($ millions) Percentage 
Transition fees 46.5 1.8 
Firm fixed fees I 2atsi 47.8 
Firm fees 808.8 30.7 
Variable fees 417.0 15.8 
Cost reimbursable fees | 103.6 Sg) 
Total 2,634.0! 100.0 


‘The contract totals $2.8 billion including the GST. 
Source: National Defence, April 2002 
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Firm fixed payments are unconditional and irrevocable 


446 A non-share, not for profit corporation was established to purchase 
equipment for the NATO Flying Training in Canada program. It issued a 
$720 million bond to obtain the capital required to buy the aircraft and 
simulators. 


441 National Defence must make 40 semi-annual payments of 

$31.4 million over 20 years (December 1999-June 2019) which go to the not- 
for-profit corporation to cover the cost of the principal and interest on these 
bonds and its operating costs. These payments are unconditional and 
irrevocable regardless of whether National Defence has the use of the aircraft. 
National Defence must pay even if the prime contractor is placed in default 
and the contract is terminated. However, should this occur, National Defence 
would continue to have access to the aircraft. 


448 In December 1999, National Defence paid $31.4 million but the 
aircraft were late in being delivered and, therefore, could not be used for 
training. National Defence cannot recover this payment despite the late 
delivery of the aircraft. There are provisions in the contract to ensure that 
National Defence retains access to the aircraft and simulators if the program 
is extended to provide the missed training. However, it is not clear at this 
point whether National Defence will have to pay some of the expenses 
necessary to keep the aircraft or whether this is the responsibility of the prime 
contractor. At the end of the program, the licence agreement states that 
National Defence can purchase the aircraft at fair market value. 


449 In addition, National Defence paid the prime contractor $15 million in 
firm and variable fees to meet its contractual obligations even though the first 
four courses had to be cancelled. The contract required that National 
Defence pay these fees as per the agreed schedule. The prime contractor has 
returned $2.5 million of the $15 million advance payment, the pre-paid 
variable components of the fees, since no students were trained. 


4.50 The contract does state that training not provided can be made 
available at some future date. The contractor is willing to provide the training 
but, until it does, it has been paid $12.5 million for training that has not yet 
been conducted. As well, problems with program delivery must be resolved 
first in order to clarify how the provisions in the contract will apply. 


National Defence has not yet determined whether it has under or over paid the variable 
fees to the prime contractor 


451 National Defence makes semi-annual prepayments for variable and 
reimbursable fees based on an estimated amount of consumption. These fees 
are to be adjusted at regular six month intervals based on actual use. To date, 
this has not been completed. National Defence and the prime contractor 
have yet to finalize the number of flying hours used. 


452 National Defence has paid $44.2 million to the prime contractor in 
variable and reimbursable fees of which almost $11 million is for international 
participants. It has not used all of the training slots and is entitled to a refund 
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for these advanced payments. National Defence is still reviewing a proposal 
from the prime contractor for a refund of $6.6 million for the training period 
from 1 January 2001 to 30 June 2001. This number remains in dispute. 


453 The contract states that the prime contractor will provide an 
information management system, but this has not yet been put in place. As a 
result there have been delays in finalizing the reconciliation, since all records 
must be reviewed and accounted for manually. In September 2001, the 
contractor and program participants agreed to change the method of 
reconciliation. National Defence personnel on site in Moose Jaw who were 
expected to provide the input for the reconciliation were not trained in the 
process. Until the reconciliation of flying hours has been finalized, National 
Defence cannot determine whether it has under or over paid the prime 
contractor for these fee elements. 


4.54 The agreements with the other countries involved with the NATO 
Flying Training in Canada program require a similar type of reconciliation at 
regular six month intervals. Since National Defence has not reconciled with 
the prime contractor, it cannot reconcile variable fees with the other nations. 
The result is that National Defence does not know if it has over or under 
charged other nations for the training that they have received. 


Payments are not tied to performance 


4.55 National Defence’s payments under the contract are based on 
schedules rather than performance milestones. In the event that the prime 
contractor is non-compliant with the terms of the contract or where the 
prime contractor does not provide the specified levels of service, there are no 
financial incentives that can be used by National Defence. Conversely, there 
are no performance incentives to reward or encourage the prime contractor 
for exceptional service. 


4.56 The only remedy available to National Defence under the contract 
would be to place the prime contractor in default of the contract. Short of 
putting the contractor in default, National Defence may not withhold 
payments regardless of the quality or quantity of the service provided. A 
National Defence study of the Alternative Service Delivery program 

(May 2001) concluded that for the NATO Flying Training in Canada 
program “the only real financial incentive [to ensure that the prime 
contractor provides full delivery of services] is the threat of termination of the 
contract.” 


457. Although the NATO Flying Training in Canada program has been 
providing training to students since June 2000, the prime contractor and 
National Defence have never reached an agreement as to how they will 
measure the performance. There is still a disagreement between the parties as 
to what constitutes availability of the aircraft. While the prime contractor 
may have the required number of aircraft operational on a particular day, 
factors such as slow turnaround of the aircraft may cause the schedule to slip 
such that it results in the cancellation of sorties. These missed sorties lead to 
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differences between the required and actual number of sorties provided and it 
also raises questions as to the level of contractual compliance. This issue has 
been outstanding since the beginning of the program and while progress has 

recently been made, there is still no agreement on what constitutes successful 
performance by the prime contractor. 


There are no mechanisms in the contract to allow for changes 


4.58 Unlike many standard government contracts, this contract does not 
include any specific clauses describing allowable ways to change the contract. 
Even though it is not possible to foresee all situations that may occur, 
particularly given the long-term nature of the agreement, we would have 
expected the contract to include a clause that would outline the procedures 
for implementing changes. The personnel of Public Works and Government 
Services Canada that are involved in the management of the contract have 
recognized the need for a change process. They have begun work on getting 
an agreement to outline a procedure for changes to the contract. 


459 In its 1999 report on Examining the Value for Money Deals under the 
Private Finance Initiative, the United Kingdom National Audit Office states 
that in long term contracts, “change is inevitable as the Department’s needs 
and priorities will alter over time.” The program does allow for some 
adjustment by selling training capacity between countries and drafting of 
memoranda of understanding, but any contract changes would involve 
re-negotiations. 


The NATO Flying Training in Canada program did not include management practices 
consistent with a program of this size and risk 


460 The government recognizes the risk associated with managing large, 
complex projects and has a policy in place to identify and manage them. 
Projects that exceed $100 million and that are assessed as high risk are 
considered major crown projects. The NATO Flying Training in Canada 
program is a complex project valued at about $150 million per year for 

20 years. Although the NATO Flying Training in Canada program exhibits 
many characteristics that are common to major crown projects, it was not 
managed as one. Treasury Board Secretariat officials told us that, at the time, 
the policy was only applied to capital acquisitions while this project was 
considered to be a service delivery contract. It was subject to a high-level of 
management reporting and oversight. 


461 Nevertheless, due to similar characteristics, we expected to find similar 
management practices as part of this program’s initial governance 
arrangements, such as an integrated risk management process and a 
performance measurement system. As a result of some of the difficulties 
realized during the first two years of implementation, Public Works and 
Government Service Canada officials have now brought in some of the 
management structures described in the Treasury Board Manual section on 


major crown projects. 


Report of the Auditor General of Canada—September 2002 Chapter 4 | 15 


NATIONAL DEFENCE—NATO FLYING TRAINING IN CANADA 


CT-155 Hawk and CT-156 Harvard || 
aircrafts. 
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462 The staffing and personnel requirements for the program were based 
on the assumption, made in the absence of a satisfactory risk assessment, that 
the contract was a routine service contract, despite its large dollar value and 
multi-year nature. The program was not given the same level of resources 
that a major crown project would receive. Public Works and Government 
Services Canada have recently hired a person to act as the local contract 
officer in Moose Jaw and provide a link between the site and the project 
office. 


The program is slow in devolving a management framework to the users 


4.63 Administration of the contract and dealings with the prime contractor 
have been centralized in the project management office at National Defence. 
The program was planned so that the operation would be producing the 
contracted levels of service by February 2002. However, given the number of 
outstanding contractual issues that have not been resolved the project 
management office will remain in place for an additional two-year period. 


4.64 The 15 Wing Moose Jaw 2002 business plan identifies the lack of a 
clear governance and accountability chain to allow timely decision making 
and effective management. Until very recently, limited information or 
decision-making authority have been devolved to the local training 
personnel, even though they are the ones who have to deal with prime 
contractor on a daily basis and are responsible for ensuring that the program 
meets the needs of the students. It also identifies constraints on their ability 
to plan and operate effectively, including shortfalls with the NATO Flying 
Training in Canada program and contract and misunderstandings with the 
prime contractor, such as 


* the command and control structure is too complicated, 
* there is not a clear understanding of responsibilities, and 


* there is a lack of visibility of available resources which has resulted in 
cumbersome and protracted planning and decision making cycles. 


4.65 Officials of National Defence and Public Works and Government 
Services Canada have recently started developing a governance and 
accountability framework as part of the NATO Flying Training in Canada 
program’s transition to the steady state level of operations. Public Works and 
Government Services Canada has created a director level position to oversee 
the project. This has assisted in bringing some major issues to the foreground. 


4.66 In an internal departmental study released in April 2002, National 
Defence identified many of the issues being reviewed in this audit, specifically 


* no single point of accountability for the program, 


* poorly defined roles and responsibilities with no person or position 
clearly responsible for dealing with unique governance issues, 


* lack of communication between all levels and slow response to local 
base concerns, 


* disjointed or non-existent business planning, 
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* lack of understanding of prime contractor responsibilities, 


* a limited number of personnel who understand the terms of the NATO 
Flying Training in Canada contract, 


lack of skilled personnel to address contracted service problems, 
* lack of succession planning, and 


* poor Air Force control over the pilot production process. 


467 The report does note some strengths of the existing program, including 
the strong central control and effective contract management performed by 
the project office, and the benefits of co-locating the technical authority, 
contract management and Public Works and Government Services Canada 
personnel within the same office. 


National Defence is looking into new management arrangements for the program and 
lessons learned 


468 There are various governance studies currently under development 
within National Defence which have identified particular concerns with the 
management and accountability framework. These governance studies come 
well into the third year of operation of the program. We expected that the 
project team would have had a plan to guide the transition to the steady state 
level of operations. Still, we are encouraged that National Defence and Public 
Works and Government Services Canada are addressing these issues at this 
time. 


4.69 Given the unique structure and the creative approaches used in the 
provision of services under this agreement, there is increased risk for the 
Department but also a significant opportunity for learning and innovation. 
Since National Defence is looking to enter into similar agreements in the 
near future with the re-tendering of flight training services at Portage La 
Prairie and other service contracts, we expect that the lessons learned from 
the NATO Flying Training in Canada program would be documented for use 
both across the department and across government. However, formal lessons 
learned have not yet been recorded on this project either by National 
Defence or Public Works and Government Services Canada. 


470 National Defence has recently started work on a major service delivery 
framework. Although this initiative is in the early stages of development, the 
aim is to provide a framework for large-scale contracts for the delivery of 
complex services. Its purpose is to provide National Defence with easy access 
to sources of information and set the stage to develop and use an integrated 
set of policies, procedures, processes, and tools. 


Procurement Reform initiatives are underway but progressing slowly 


4.71 In our 1999 Report, we recommended that, as part of its on-going work 
on procurement reform, guidelines and training be developed for large multi- 
year service contracts and that these should address key issues of how 
competition is to be addressed where long-term “partnering” would be 
beneficial to the government. In 1999, the Treasury Board Secretariat 
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indicated that they were developing a guide and a database on alternative 
service delivery. The Treasury Board Secretariat was also leading an 
interdepartmental initiative on procurement reform, which would include 
guidance for large, multi-year service contracts. Public Works and 
Government Services Canada also indicated that they would be working with 
the Treasury Board Secretariat in this regard. 


472 In our December 2001 follow-up on alternative service delivery we 
reported that the Treasury Board Secretariat, Public Works and Government 
Services Canada and National Defence were working on a framework and 
best practices guide on long-term complex contracts. They have recently 
completed a paper on Strategic Direction and Best Practices in Complex 
Procurement. The paper documents the strategies and techniques that have 
been developed by Public Works and Government Services Canada and 
National Defence on dealing with an increasingly complex procurement 
environment and Treasury Board Secretariat officials said that it will be used 
as a guide for future procurement. 


4.73. In April 2002, the Treasury Board Secretariat released a new Policy for 
Alternative Service Delivery. While the new policy provides some limited 
guidance on the governance and performance monitoring for alternative 
service delivery projects, it provides little direction for long term, large dollar 
value service arrangements. The planned alternative service delivery 
database is not yet operational. 


4174 Public Works and Government Services Canada is developing a 
training course covering complex procurement. Department officials 
informed us that the program has been piloted and is nearing completion of 
the development phase. 


Conclusion and Recommendations 


415 National Defence did not train the contracted number of student pilots 
for which it paid in the NATO Flying Training in Canada program. During 
the first two years of this program it used only 40 percent of the basic Phase 
IIA flying training slots, 28 percent of the basic Phase IIB training, 50 percent 
of the advanced Phase III training and 58 percent of the Phase IV fighter 
lead-in training for which it has contracted. We estimate that as of the end of 
December 2001, the Department had paid about $65 million more than the 
value of training received. This amount is growing and will continue to grow 
as long as the program remains underutilized. 


476 National Defence recognizes that it has had program management 
problems. Studies are underway to determine how best to manage the NATO 
Flying Training in Canada program but there are outstanding issues that must 
be resolved. Developing a solid governance arrangement is the first step in 
improving overall management. 
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4.11 Recommendation. The Department should resolve the program 
management issues and implement a revised management framework as a 
matter of urgency. 


National Defence’s response. The Department will continue to evolve the 
governance structure to meet complex needs and a steady state structure will 
be put in place over the next year. NATO Flying Training in Canada is a 
unique and complex program being managed in a constructive and 
evolutionary fashion. There has been significant oversight at senior 
management levels from two government departments during the difficult 
start-up period. Now responsibilities are being slowly, carefully, and 
appropriately delegated to various levels within both the Department and 
Public Works and Government Services Canada. Although there has been 
much discussion and some differences of opinion over the program’s steady 
state governance structure, we maintain that identifying and then resolving 
issues reflects an appropriate management approach. 


478 We have reported on underused training capacity in the past and have 
concerns that problems may be repeated as new contracts are negotiated. 
This program does not have the flexibility needed in long-term contracts to 
accommodate changing needs over time or the measures to ensure 
performance meets expectations. 


479 Recommendation. Lessons learned from the NATO Flying Training in 
Canada program should be documented and reported to senior management 
with an action plan. 


National Defence’s response. The Department agrees that formal 
documentation of lessons learned is important. Moreover, lessons learned in 
the NATO Flying Training in Canada Program are already being applied in 
real-time to the Contracted Flying Training and Support project that is being 
managed within the same project office. The Department’s intent is to 
formally document all applicable lessons this fiscal year and then promulgate 
them through appropriate media. 


480 Recommendation. New contracts of a similar nature to the NATO 
Flying Training in Canada program that the Department enters into should 
ensure that 


* payments are tied to performance and value received, and 


* a project management framework commensurate with the risk and 
design of the program is in place at the outset. 


National Defence’s response. It is unlikely that many of the unique risk- 
sharing features of the NATO Flying Training in Canada contract will arise in 
the foreseeable future. However, as was the case with this particular contract, 
the Department will continue to tailor project management structures and 
contract terms and conditions to the unique circumstances of each 
contractual arrangement. 
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481 Recommendation. The Department should ensure that the services 
purchased through a long term service contract are aligned with the ability to 
use them. 


National Defence’s response. The Department agrees that this is an 
appropriate goal but disagrees with the implication that this was not the case 
with NATO Flying Training. We believe that the under utilization of capacity 
observed by the Office of the Auditor General was a reflection of the 
challenges inherent in starting up a new program of this magnitude and 
complexity. The Department remains confident that purchased training 
capacity will be more fully utilized in future as the program matures. In the 
NATO Flying Training program, the possibility of reduced training demand 
over time was considered and there is a contractual provision for Canada to 
sell off 50 percent of our “jet pilot” training capacity to other nations. Further, 
since the NATO Flying Training in Canada program was designed to be 
marketed, there is a built-in marketing infrastructure to dispose of non- 
required capacity should that occur. 


National Defence’s comments. The NATO Flying Training in Canada 
Program is a unique program of risk-taking and risk-sharing, which, despite 
some start-up hurdles that are being overcome, represents a huge leap 
forward in training technology and training philosophy. The Department 
agrees that training production in Phase II has been less than expected in the 
first two years, but asserts that any missed training will be made up later in the 
contract. The contractor has an obligation to make up all the contracted 
training over the life of the program and the contract is being enforced. 


This program represents the best long-term option for the training of 
Canada’s military pilots. NFTC is a success with over $1 billion of training 
sold internationally. NFTC is now the recognized benchmark against which 
future pilot training systems are being compared. 
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About the Follow-Up 


Objectives 
This follow-up audit had two objectives: 


* to examine the implementation and management of the NATO Flying Training in Canada program and to 
determine whether the service contract was meeting the needs of the department in an economical and 
efficient way, and 


* to review whether the Treasury Board Secretariat and Public Works and Government Services Canada had 
made any progress in responding to our recommendations about procurement reform and “partnering” as stated 
in the 1999 chapter on alternative service delivery chapter. 


Scope 


Our audit focussed on the management framework and processes with respect to the NATO Flying Training in 
Canada program in place at National Defence, the Canadian Forces, 1 Canadian Air Division and the Canadian 
Aerospace Training Program office. In addition, it looked at the management and delivery of contracted services at 


15 Wing, Moose Jaw, and 4 Wing, Cold Lake. 


We assessed the implementation and management of the NATO Flying Training in Canada program from the time 
the contract was signed in May 1998 until March 2002. This represents a very short period in the overall life of the 
20-year contract worth $2.8 billion. While our audit focussed on this time period, we have made certain comments 
about the way the planning was done in the pre-contract stage. 


We also focussed on work done to date by the Treasury Board Secretariat and Public Works and Government 
Services Canada on implementing our 1999 recommendations. 


We did not audit the activities of the prime contractor and, as a result, do not comment on them. We do bring 
forward facts regarding their activities but limit our comments to the whether the actions of government officials 
were appropriate or not. 


Criteria 


We followed up on the recommendations for procurement reform made to the Treasury Board Secretariat and Public 
Works and Government Services Canada in our 1999 audit on alternative service delivery. 


We expected that National Defence would ensure that the goods and services contracted for would be delivered as 
stated in the contract. We expected that National Defence would act in a way compatible with the Financial 
Administration Act to ensure that only money earned is paid. 


We also expected that the Canadian Forces would be able to train the number of pilots required as indicated in the 
contract in a cost-effective way. 


Ratings 
We assessed the action of departments/agencies against our original audit recommendations (see Key Message at the 
beginning of the chapter). We used the following ratings: 

* Completed. Corrective action has been fully implemented. 

: Satisfactory progress. Progress is being made at a satisfactory pace. 

+ Limited progress. Some progress is being made, but the pace or scope is not satisfactory. 


- No progress. No evidence of progress although the department or agency accepted the recommendation from 
the original audit. 


+ Rejected. The department or agency did not accept the recommendation from the original audit. 


* Unknown. Status of progress is unknown or information is not available. 
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Appendix A Excerpts from our 1999 Report, Chapter 27, National Defence—Alternative Service Delivery 


Background 


The NATO Flying Training in Canada (NFTC) program is a military pilot training program conducted in co-operation with 
industry for the Canadian Forces and other participating nations. Since it was initiated prior to the implementation of the 
Alternative Service Delivery program, it is not being managed under that framework. 


In addition to developing a cost-effective pilot training program, NFTC is supposed to achieve a number of other benefits: 
* creating employment; 
¢ keeping the base at Moose Jaw open; 
* demonstrating the capabilities of Canada’s aerospace industry; and 
¢ making a significant contribution to NATO. 


Officials also informed us that National Defence had insufficient funds to renew its training aircraft fleets. One ways to 
combat the “rust out” of the existing fleet was to implement a service contract that required annual installment payments. 


In 1996, National Defence obtained Cabinet approval of a 20-year, $2.8 billion sole-source contract with Bombardier 
Inc. for the program. Milit-Air Inc. will purchase the planes, flight simulators and other equipment with the proceeds of a 
$720 million bond. By way of a separate agreement, Milit-Air Inc. has leased the equipment to Bombardier Inc. 


About $1.3 billion of the contract funds will acquire flight simulators and a new fleet of 42 military training aircraft to 
replace National Defence’s existing fleet of Tutor aircraft that, according to departmental studies, could have been 
refurbished and made to last until the year 2015. The remaining $1.5 billion will be used by Bombardier Inc. to maintain 
the aircraft and the simulators, manage the base in Moose Jaw, and provide ground school instructors. National Defence 
will provide the overall management of the NFTC program and the flight instructors. The first flight instructors were to 
Start training in the third quarter of 1999. 


The contract was awarded without competition 


We found that the decision to award the contract without following the normal bid solicitation process for government 
contracting was not adequately justified. 


The profit mark-up in the NFTC contract is not consistent with current guidelines or supported by 
adequate analysis of contractor’s risks 


In the event of a sole-source contract, Public Works and Government Services Canada’s profit policy and guidelines are 
supposed to establish the level of profit awarded to a contractor. Public Works and Government Services Canada officials 
could not provide us with the detailed calculations and risk assessments they used to arrive at the profit markup included 
in the contract. According to officials of both departments, the NFTC program will provide the government with valuable 
benefits through the transfer of significant risks to the contractor over the next 20 years. Although departmental 
documents show the departments’ estimate of contractor risk to be between $360 million and $460 million, they had no 
calculations to support this. They estimate that the risk exposure to the contractor relates to the following: 


¢ the quantity and adequacy of the aircraft required for the program; 
¢ future increases in aircraft and infrastructure operating costs; 

¢ failure to obtain the expected number of foreign participants; and 
* environmental risks. 


We expected that the two departments would comprehensively assess the risks being transferred to the contractor and 
' estimate the value to the Crown of that risk transfer. We were unable to establish that this had been done. 


Public Works and Government Services Canada informed us that it hired an outside consultant to review the risk elements 
in this program. We note that the consultant could not perform a comprehensive review of all the risks since, at the time, 
the agreements had not been finalized. Therefore, in our opinion the review was not sufficient to provide assurance that 
there is an equitable sharing of risks under the contract. 
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In response to our audit, National Defence officials prepared a risk summary in late September 1999 that attempted to 
quantify the value of the risks transferred to the contractor. While this summary clearly identified the major risks, it did 
not assess their probability of occurrence and their overall financial impact. As a result, it is not possible to establish the 
correlation between the value of the risks and the profit markup that was negotiated in the NFTC contract. 


It should be noted that if the NFTC program expands beyond its current level, the Department is committed to paying for 
the additional aircraft and equipment that will be required. These costs would be recovered from the additional revenues 
from foreign participants. According to National Defence officials, if the program were to expand beyond its current 
capacity there would be significant financial benefits to both the Crown and the contractor, because the fixed costs of the 
program would be shared among a greater number of participants. 


The chosen financing arrangements increase some risks 
The NFTC program is the first example of “innovative” financing for a major National Defence capital project. 


The Department of Finance had suggested in late May 1997 that the Department consider purchasing the equipment 
directly and supplying it to the contractor as government-supplied equipment. In response, National Defence prepared an 
analysis of industry financing compared with government financing of the NFTC assets. We found that this analysis was 
not complete and that it was performed at a point when it was impractical to make any changes to the financing 
arrangements. 


The unique financing arrangements are also causing problems with the acquisition of the Raytheon T6-A aircraft and 
related technical data. The U.S. Department of State has serious concerns about a private company, Milit-Air Inc., owning 
military aircraft. It is concerned about Canada’s ability to control the transfer of information and the use and resale of 
aircraft owned by Milit-Air Inc. The two governments have been working on a solution, and it is expected that the 
Canadian government will be providing the necessary assurances shortly. However, the issue is not yet completely 
resolved. 


The fact that these additional risks are present leads us to believe that a more rigorous assessment of alternatives for 
acquiring the assets ought to have been prepared, and earlier in the process. 
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Appendix B National Defence’s pilot training process 


Non-NFTC* flying training 
¢ Primary flying training. 
e Conducted at Southport—Portage la Prairie, Manitoba. 
* Successful candidates progress to NFTC basic flying training IIA. 


| 


NFTC basic flying training Phase IIA 
¢ Common basic flying training is provided on the Harvard II aircraft for jet, multi-engine, and helicopter pilots. 
* Training is located in Moose Jaw, Saskatchewan. 


* Successful pilots chosen to fly multi-engine or heticopters are sent to Portage La Prairie for further training (not part of the 
NFTC program) while those chosen to fly CF—18 proceed to basic flying training IIB. 


¢ The training is planned to take 20 weeks. 


NFTC basic flying training Phase IIB Non-NFTC flying training 
¢ Training is conducted on the Harvard II aircraft at e Multi-engine and helicopter. 
Moose Jaw, Saskatchewan. * Training conducted at Portage la Prairie, 
* Successful Candidates proceed to NFTC advanced Manitoba. 
flying training III. * Students given specialized instruction on multi- 
¢ Training is planned to take 10 weeks. engine planes and helicopters. 


¢ Successful candidates are granted their wings and 
forwarded to an (OTU) Operational Training Unit 
for mission-specific training. 


NFTC advanced flying training Phase III 

¢ Training is conducted on the Hawk jet aircraft at 
Moose Jaw, Saskatchewan. 

¢ The aim of this phase is to develop advanced 
flying skills. 

¢ Successful candidates progress to NFTC fighter 
lead-in training IV. 

¢ Training is planned to take 22 weeks. 


| 


NFTC fighter lead-in training Phase IV pi pomlene Progra tae af Be to 
° ining i iat ai an Operational Training Unit for instruction in 
aes eee Se on et ‘mission specific flying’ prior to being posted 
sie AS At a : , to an Operational Unit for CF—18, Hercules, 
* The aim of this phase is to develop tactical flying Airbus, Buffalo, Aurora, Dash8, Twin Otter, 
skills and judgement, advanced jet flying, and Griffon, Sea King, Labrador, or Cormorant. 
advanced combat manoeuvres. 


¢ Successful candidates progress to the CF-18 
operational training unit. 


¢ Training is planned to take 16 weeks. 


*NFTC—NATO Flying Training in Canada 
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Appendix C NATO Flying Training in Canada program milestones 


UG )sy2 


NATO requirement identified for a common NATO fast jet pilot training program. 


Milestones 


1992-1994 National Defence begins analyzing and defining future pilot training options. 
1994 Bombardier submits unsolicited proposal to National Defence for fast jet pilot training. 


December 1994 


Bombardier submits a business case and National Defence adds NATO Flying Training in Canada (NFTC) as an 
option. 


May 1995 


Canada submits proposal to host the NFTC program to NATO. 


January 1996 


Bombardier delivers an industry proposal to National Defence. National Defence compares the options and 
identifies NFTC as the preferred option. 


June 1996 National Defence gets approval to enter into a 20 year $2.8 billion sole source contract with Bombardier to 
provide support to NFTC. Approval was given on the condition that international nations participate as a means 
of reducing costs for Canada. 

Canadian proposal to NATO nations. 
April 1997 NFTC launch and negotiations. 
May 1998 Services agreement in support of military pilot training signed between Bombardier and Public Works and 


Government Services Canada (on behalf of Canada). Moose Jaw and Cold Lake transition begins. 


September 1998 


Memorandum of understanding (MOU) signed between Canada and Denmark for participation in the program. 


July 1999 


MOU signed between Canada and the U.K. for participation in the program. 


December 1999 


Canada makes the first of 40 semi-annual payments of $32 million for use of equipment plus first instalment of 
firm, variable and cost reimbursable fees. 


Planned start date for Phase II instructor pilot training and course validation delayed due to late delivery of 
Harvard II aircraft. 


February 2000 


| First four basic flying training Phase IIA courses cancelled. 


March 2000 MOU signed between Canada and Italy for participation in the program. 
MOU signed between Canada and Singapore for participation in the program. 
June 2000 Basic flying training Phase IIA courses begin. 


December 2000 


Advanced Phase Ill flying training courses begin. 


March 2002 


Fighter lead in Phase IV flying training courses begin. 


MOU signed between Canada and Hungary for participation in the program. 
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Industry Canada 


Management of the Canada Small Business 
Financing Program 


Assistant Auditor General: Richard Flageole 
Responsible Auditors: Richard Gaudreau and Chantal Thibaudeau 


Key Message 


5.1 Industry Canada has made progress in addressing some of the issues raised in our 1997 Report, as outlined in 
the table below. We are still concerned about the ability of the Department to achieve its cost recovery objective for 
this Program. At 31 March 2001, losses on loans guaranteed between 1995 and 1999 amounted to $155 million, and 
we estimate that they will reach at least $200 million. There are also indications that disbursements on claims will 
exceed revenues for loans guaranteed after 1999. Industry Canada must develop a better model to forecast the 
financial performance of the Program. Parliament must also be informed on a timely basis of any anticipated losses. 


ORIGINAL ISSUES 


Measuring program performance 


5.2 The Department should clearly define the expected 
results of the Program and obtain relevant information on 
the Program’s results. 


5.3. The Department should ensure that the Sma// 
Business Loans Act eligibility requirements and 
conditions lead to the results expected for the Program. 


PROGRESS 


The Department developed a program evaluation 
framework in 1998 and used it to guide its data 
collection activities. However, it still needs to define 
criteria for assessing the Program’s success, including a 
target range for incrementality. 


Program eligibility criteria and conditions were examined 
as part of the review of the Sma// Business Loans Act in 
1998 and changes were made to Program conditions. 
The impact of these changes on the Program’s expected 
results, including cost recovery, will need to be carefully 
assessed. 


Cost recovery 


5.4 We expressed uncertainty about whether full cost 
recovery would be achieved. We concluded that the dual 
objectives of increasing access to financing while 
recovering the costs of the Program called for careful 
analysis. The Department should carefully monitor the 
performance of the loans portfolio and improve systems 
and practices for forecasting the future performance of 
the Program. 


= eee ee Nes 


The Department considers that it will not meet its cost 
recovery objective for loans guaranteed between 1995 
and 1999. We estimate that disbursements on claims 
for that period will exceed revenues by at least 

$200 million. There are also indications that the 
Department may not meet its cost recovery objective for 
loans guaranteed after 1999. 


The Department now has good information on portfolio 

risks. This provides a basis for monitoring the financial 

performance of the Program. The Department has yet to 
adjust its claims forecasting model. 


RATING* 


LIMITED 
PROGRESS 


LIMITED 
PROGRESS 


LIMITED 
PROGRESS 


*Possible ratings are: completed, satisfactory progress, limited progress, no progress, rejected, unknown. (See About the Follow-Up for an explanation of 


the ratings.) 
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PROGRESS RATING* 


ORIGINAL ISSUES 


Improving the Program’s delivery 


Legislation was amended to reinforce the principle of 
due diligence by financial institutions and to give the SATISFACTORY 
Minister authority to conduct compliance audits. The PROGRESS 

Department is working on an audit strategy, and it plans 
to start conducting audits in the fall of 2002. 


5.5 The Department should ensure that lenders have 
exercised due care in making loans and have complied 
with the Small Business Loans Act and Regulations. 


The Canada Small Business Financing Act clarifies this 
matter, and the Department now requires the borrower 
to declare that it has not exceeded the maximum loan 
limit. 


5.6 The Department should assess the need to limit 
access to loans by related entities to the maximum 
amount allowed and, if the need exists, seek amendment 
to the Sma// Business Loans Act. 


COMPLETED 


5.7 The Department should take the necessary 
measures to reduce the amount of interest paid on 
claims made by lending institutions. 


The Department has introduced two main measures: the 
interim payment of a claim before the borrower’s 
personal collateral has been collected and the non- COMPLETED 
payment of interest beyond 24 months following the 
date of the loan default. 


Accountability to Parliament 


9.8 The Department should ensure that The Department has improved information on cost LIMITED 
parliamentarians are provided with the necessary recovery but it provides limited information on the PROGRESS 
information to assess the extent to which the Program is | expected financial performance of the Program and 

managed efficiently and is achieving its objectives. efficiency of operations. 


5.9 The Department should report information on job The Department has clarified the meaning and indicated SATISFACTORY 


creation that reflects actual results attributable to the some limitations of the information provided on the 
3 PROGRESS 
Program. number of jobs to be created by the Program. 


5.10 Since 1998 there has been a significant decrease in the number of loans made through the Program. The Department should 


continue its efforts to determine the reasons for the drop and how they affect the rationale for the Program and its parameters. It 
should also investigate the concerns that financial institutions have expressed about the administrative requirements of the Program. 


*Possible ratings are: completed, satisfactory progress, limited progress, no progress, rejected, unknown. (See About the Follow-Up for an explanation of 
the ratings.) 


Industry Canada has responded. The Department has generally agreed with our conclusions. Its response, 
presented at the end of the chapter, elaborates on the action it intends to take to address both the issues we raise in 
the report and our recommendation. 
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Introduction 


5.11 In Chapter 29 of our December 1997 Report, we reported on Industry 
Canada’s management of the Small Business Loans Program, governed at the 
time by the Small Business Loans Act. 


5.12 In that chapter, we said that Industry Canada needed to define more 
clearly the results it expected the Program to achieve and better measure its 
performance toward achieving them. We questioned whether the 
Department could meet its objective of full cost recovery, given the fee 
structure and loss-sharing ratio then in effect. We emphasized the need to 
carefully analyze the dual objective of making loans more accessible while 
recovering the costs of the Program. We stressed that Industry Canada 
needed tighter controls to ensure that lenders exercised due diligence in 
granting loans. We said it should take appropriate steps to reduce the 
amounts it was paying to financial institutions in interest on their claims. 
Finally, we noted that Parliament needed better performance information on 
the Program. 


Events since our audit of 1997 


§.13 In 1998, the Standing Committee on Public Accounts held hearings on 
our report and made recommendations similar to ours in its own report to 
Parliament. The House of Commons Standing Committee on Industry and 
the Standing Senate Committee on Banking, Trade and Commerce each held 
public hearings on the Program that same year to better understand the 
financing needs of Canadian small and medium-sized enterprises (SMEs). 


5.14 These reviews led to the Canada Small Business Financing Act, which 
came into force on | April 1999 and replaced the Small Business Loans 
Program with the Canada Small Business Financing Program. The new 
legislation maintains the fundamental objective of the Program and its key 
parameters (percentage guaranteed by the government, type and size of 
business, and registration and administrative fees). It also provides for a 
mandatory comprehensive review of the Program every five years; the next 
review report must be tabled in Parliament in 2004-05. It reinforces the 
principle by which financial institutions must show due diligence in managing 
loans registered under the Program, and it gives the Minister authority to 
conduct on-site audits to verify their compliance. Exhibit 5.1 provides 
additional information on the history of the Program. Exhibit 5.2 provides 
data on the value of loans guaranteed over the past 10 years. 


The program today 


5.15 The purpose of the Canada Small Business Financing Program is to 
increase the availability of financing to establish, expand, modernize, and 
improve small business enterprises. Industry Canada is committed to 
compensate financial institutions for 85 percent of their eligible losses on 
loans they have made under the Program. Any business (except for religious, 
not-for-profit, and farming organizations) with gross annual revenues below 
$5 million is eligible for financing to a maximum of $250,000. The maximum 
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repayment period for a loan under the Program is 10 years. A lender is 
allowed to obtain a personal or a corporate guarantee from the borrower, 
although a personal guarantee cannot exceed 25 percent of the loan amount. 
Businesses can use the loans to finance real property and buildings, leasehold 
improvements, or equipment. Exhibits 5.3 and 5.4 show loans guaranteed 
under the Program in 2000-01, by industry sector and by area of use. 


Exhibit 5.1 Program history 


From 1961 until 1993, the Small Business Loans Program was stable and relatively 
modest. Between 1990 and 1993, the average annual value of loans guaranteed 
under the Program was $438 million, and the Department paid financial institutions 
an average of $41 million a year in claims and interest for losses on loans they had 
made under the Program. 


In 1993, a number of changes to the Sma// Business Loans Act broadened the 
eligibility criteria for the Program by raising from $2 million to $5 million the maximum 
annual revenue of a business defined as “small business.” Also raised were the 
maximum loan amount, the portion of a loan that the Department would guarantee, 
the percentage of a project’s total funding allowable under the Program, and the 
registration fees. 


The result was more loans, for higher amounts. The annual value of loans guaranteed 
under the Program rose from $502 million in 1992-93 to $2.5 billion in 1993-94 
and $4.4 billion in 1994-95. This unprecedented increase created concern about the 
financial consequences for the government: very likely, it would see a corresponding 
increase in claims by financial institutions for losses. In fact, by 31 March 2002, loans 
guaranteed between 1 April 1993 and 31 March 1995 had generated $527 million in 
claims, which the government paid. We estimate that the actual loss suffered by the 
government was $388 million after taking into account the registration fees it had 
received for the same loans. 


That concern led Parliament in 1995 to tighten certain of the 1993 amendments to 
the Act that had broadened eligibility for the Program, and the Program’s major 
parameters have remained the same since then. 


Exhibit 5.2 Annual value of loans guaranteed from 1990-91 to 2000-01 


$ billion 
5 


1990-91 1991-92 1992-93 1993-94 1994-95 1995-96 1996-97 1997-98 1998-99 1999-2000 2000-01 


Source: Industry Canada 
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Exhibit 5.3 Loans guaranteed though the Program to industry sectors in 2000-01 


All other sectors Accommodation, food, 
28% and beverage services 
23% 
Other services Retail trade 
15% 14% 


Manufacturing Transportation and 
11% warehousing 
9% 


Source: Industry Canada 


Exhibit 5.4 Areas of use of loans guaranteed through the Program in 2000-01 


Registration fees 


1% 
| 
Software Bre p AOE 
1% : 
Leasehold improvement 

F 18% 
Equipment 
59% 


Source: Industry Canada 


§.16 Industry Canada delivers the Program in partnership with financial 
institutions. A small business applies to an institution for financing; the 
institution assesses the application and assumes responsibility for deciding 
whether or not to grant the loan. If it does, it makes the loan directly to the 
applicant. It is responsible for all aspects of credit management. 


5.17. When a financial institution determines that a loan meets the 
conditions of the Program, it submits it to the Department for registration as 
a guaranteed loan with a 2 percent registration fee it has collected from the 
borrower. An annual administration fee of 1.25 percent based on the average 
monthly loan balance is also charged and turned over to the Department. 
Lenders cannot charge more than 3 percent above the prime lending rate on 
floating-rate loans or more than 3 percent above the residential mortgage rate 
on fixed-rate loans; the interest charges include the 1.25 percent 
administration fee. If the borrower defaults, the financial institution must 
collect the security pledged against the loan before it can submit a claim to 
Industry Canada. 
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Measuring program performance 


5.18 The Small Business Loans Administration Directorate of the 
Department’s Operations Sector administers the Program under the 
provisions of the Act and its Regulations. The Directorate has an annual 
operating budget of $3 million and a staff of 30 people, most of them assigned 
to claims processing. The Small Business Policy Branch, part of the 
Department’s Policy Sector, provides strategic direction for the financing of 
small businesses. 


5.19 In 2000-01, more than 14,000 loans with a value of about $1.2 billion 
were registered under the Program. The average amount of each loan was 
about $80,000. In the same period, the Department paid out about 

$171 million in claims for loans guaranteed under the previous and the new 
legislation. 


Focus of the follow-up 


§.20 Our follow-up looked at Industry Canada’s management of the Canada 
Small Business Financing Program and its administration of the Canada Small 
Business Financing Act and the Small Business Loans Act, which guide the 
Program. The purpose of our follow-up was to assess Industry Canada’s 
progress in addressing the issues raised in our 1997 audit and to determine 
whether the Department now has in place the systems and practices to 
manage the Program efficiently, cost-effectively, and in accordance with 
legislative authority. Our follow-up scope and criteria are set out in About the 
Follow-Up at the end of the chapter. 


Observations and Recommendations 


5.21. According to Industry Canada, investment stimulates economic 
growth and innovation and helps ensure sustainable development; one of the 
Department’s strategic objectives is to promote investment in Canada. It 
views limited access to capital as a barrier to investment by small and 
medium-sized enterprises (SMEs). 


5.22 The intent of the Canada Small Business Financing Program is to 
encourage financial institutions to give small businesses greater access to 
financing for their creation, expansion, modernization, and improvement. 


§.23 Our 1997 report recommended that Industry Canada clearly define the 
results it expected the Program to deliver and obtain relevant information on 
the results it achieved. We also stressed the need to ensure that the Program’s 
eligibility requirements and conditions foster the achievement of the desired 
results. After examining our report in 1998, the Standing Committee on 
Public Accounts recommended that the government clearly define its 
expectations for the Program’s performance, set clear targets for 
incrementality, establish appropriate indicators of performance, and ensure 
that the Program met the financing needs of small businesses. 
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A performance evaluation framework is in place 


5.24 In 1998, the Department developed a performance evaluation 
framework for the Program. This evaluation framework, developed as part of 
the review of the Small Business Loans Act, identifies evaluation issues, 
performance indicators, data collection requirements, and options for the 
future evaluation of the Program. Issues to be evaluated are the relevance of 
increased access to financing for small business and the need for federal 
involvement; the level of incrementality achieved by the Program; progress 
toward cost recovery; reliability of claims forecasting; the Program’s impact 
on job creation, maintenance, and displacement; awareness of the Program; 
and the performance of borrowers. 


Data collection on the Program’s performance has begun 


5.25 In 1998 the Task Force on the Future of the Canadian Financial 
Services Sector (MacKay Task Force) found that the development of a public 
policy on the financing of small and medium enterprises (SMEs) was hindered 
by a lack of data. It recommended that the government make a concerted 
effort to improve the quality and quantity of information on SME financing. 
In response, Industry Canada, Statistics Canada, and the Department of 
Finance Canada launched the SME Financing Data Initiative. The 
Department has carried out six studies under the initiative to better 
understand the issue of SME financing and plans to continue studying it over 
the next few years. The studies provide valuable information on the relevance 
of the program. 


5.26 To measure the effectiveness of the Program, the Department has 
conducted a survey of businesses to assess the impact on both the job creation 
and the performance of businesses that had received loans. It also surveyed 
SMEs to better understand their experience in looking for and obtaining 
financing and to determine how much they knew about the Program 


Still a need to define criteria for assessing the Program’s success 


5.27. In 1997, we indicated that it was important for the Department to 
define the level of incrementality it expected the Program to achieve, and 
adjust the parameters of the Program accordingly; the Standing Committee 
on Public Accounts made the same recommendations in 1998. (The level of 
incrementality is the extent to which businesses obtain financing through the 
Program that otherwise would not be available to them or would be available 
on less favourable terms.) The evaluation framework developed by the 
Department still does not specify the rate of incrementality that would 
represent success. The latest related study by the Department, in 1996, found 
that 54 percent of the loans guaranteed through the Program could not have 
been obtained without it. 


5.28 The Department is presently reviewing its 1998 evaluation framework 
to take into account Treasury Board Secretariat’s new policy on measurement 
of results. The review is intended to lead to a new results-based management 
and accountability framework. According to the Department, the new 
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Cost recovery 


framework will be in place by the fall of 2002 and will guide the evaluation of 
the Program in 2004. 


The Department is unlikely to achieve its cost recovery objective 


5.29 Under the cost recovery policy adopted in 1995, the Department aims 
to balance the Program’s costs (claims and interest it pays to financial 
institutions) with its revenues (registration and administration fees). The 
Program costs in this equation exclude the Department’s salary and operating 
expenses for the Program. 


5.30 In 1997, we questioned whether the Department could achieve cost 
recovery given the fee structure and loss-sharing ratio then in effect. In this 
follow-up, we reviewed the Department’s data and reports on the Program’s 
financial results for loans guaranteed between April 1995 and March 1999 
under the Small Business Loans Act and between April 1999 and March 2002 
under the Canada Small Business Financing Act. 


§.31 Financial performance—1995 to 1999. In its 2000-01 annual report 
on the Program, the Department stated that it does not expect to meet its 
cost recovery goal for loans guaranteed between | April 1995 and 

31 March 1999. It considers that cost recovery can be achieved only if paid 
claims do not exceed 6.25 percent of the value of guaranteed loans, and it has 
indicated that claims to be paid on loans guaranteed during that period will 
amount to 9 percent of the total value of the loans. At the end of 

March 2001, this represented losses of $155.8 million. We estimate that these 
losses will exceed $200 million over the life of the loans. 


5.32. Financial performance—1999 to 2002. When it reviewed the 
Program in 1998, the Department considered that no changes were needed in 
the Program’s key parameters (percentage guaranteed by the government, size 
and type of business, and registration and administrative fees structure). It did 
recommend changes, however, that it believed would allow for a better 
balance between revenues and expenditures. Among the changes were an 
obligation for lenders to exercise the same diligence as they do with their 
conventional loans; the possibility for the Department to conduct on-site 
audits of lenders’ files; new provisions to prevent loan splitting; interim 
payment of claims to reduce interest costs; and a broadening of the rules on 
security pledged against loans made under the Program. These recommended 
changes were incorporated in the revised legislation. 


5.33 A study conducted for the Department in November 2001 on payment 
defaults and claims for loans registered from 1995 to 1998 identified the 

sectors with the highest loss rates. The rate for the accommodation, food, and 
beverage industry was 11.6 percent, for leasehold improvements 9.7 percent, 
and for franchising 9.6 percent. The study also showed that loans of $100,000 


or more also carried a higher loss rate, at 14.4 percent. 


5.34. Our review of the distribution of loans guaranteed since 1999 shows 
that with one exception, the same sectors represent an increasing share of the 
portfolio, suggesting that the overall level of risk is increasing as well: 
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* The proportion of total loan value that went to the accommodation, 
food, and beverage industry increased from 13.1 percent to 
22.¢ percent. 


¢ The proportion of total loan value that went to the leasehold 
improvements sector increased from 14.2 percent to 17.5 percent. 


* Loans of $100,000 or more increased from 21.6 percent to 27.8 percent 
of the total number of loans. 


The exception is the franchising sector, whose share of the total loan value 
has decreased from 12 percent to 11 percent over the same period. 


5.35 Our analysis of claims paid to date on loans guaranteed in 1999-2000 
suggests that there is no downward trend in the loss rate. Claims paid on 
loans registered in 1999-2000 amount to 3 percent of the value of the loans. 
This three-year loss rate is similar to three-year loss rates observed since 
1995-96. We believe it is unlikely that the rate of claims paid will decline 
enough to allow for cost recovery. The Department is still faced with the 
difficult objective of achieving cost recovery while maintaining an acceptable 
rate of incrementality. 


Monitoring capacity is improving; forecasting methods need to be enhanced 


5.36 In 1997, we recommended that to improve its chances of meeting its 
cost recovery objective, Industry Canada monitor changes in the performance 
of its loan portfolio more closely and consider any new economic factors that 
could prevent cost recovery. We also recommended that it enhance the 
systems and practices it used in forecasting how well the Program would 
perform. We believed that the Department needed to analyze its loan 
guarantee portfolio regularly for risks inherent in characteristics such as 
industry sector, type of loan, region, and age of the business. 


5.37 The November 2001 study of payment defaults and claims for loans 
registered from 1995 to 1998 allowed the Department to understand better 
the factors contributing to increased risk in its loan portfolio and to 
determine the actual financial performance of these loans. The Department 
had last carried out such a study in March 1997. This provides a basis for 
future regular monitoring of the Program’s financial performance. Also, the 
Department’s annual report on the Program now includes the level of claims 
paid, by industry sector; and the net cost of the program, by year in which the 
loans were made. 


5.38 The Department has had a model to forecast the financial performance 
of the loan portfolio since 1994; responding to our 1997 recommendation, it 
said it would revise the model. In this follow-up, we observed that the 
Department has not made the changes that would allow its forecasting model 
to better reflect factors that could impact the level of claims. We still believe 
that a better forecasting model would give the Department an effective tool 
to ensure prudent management of public funds and inform Parliament of 
anticipated shortfalls. 
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Improving the Program’s delivery 


Tools being developed to monitor compliance with the Act and Regulations 


5.39 In 1997 we examined files of financial institutions on loans made under 
the Small Business Loans Act, and we observed that they did not always 
contain evidence that the Program rules had been followed. We stressed that 
the Department needed mechanisms to ensure that financial institutions 
were exercising due care when they issued loans. 


5.40 The changes made to the legislation in 1999 reinforce the principle 
that financial institutions must exercise due diligence in making and 
administering loans under the Program. The Regulations now require that 
lenders apply the same procedures to these loans as they would to their 
conventional loans. 


5.41 The new Act also gives the Minister the authority to audit for 
compliance with the Act and Regulations, including the due diligence and 
documentation requirements. The Department is finalizing an audit strategy 
for on-site review of practices used by financial institutions. It has not yet 
started conducting audits; it has informed us that it will begin in the fall 


of 2002. 


The issue of related entities has been clarified 


5.42 We noted in 1997 a number of separate corporations with substantially 
common ownership had obtained among them more than $250,000 in loans 
to operate the same business. We said that situations like this increased credit 
risk and could cause higher losses in the loan portfolio. We also pointed out 
that the Act contained no provisions to prevent a group of related entities 
from obtaining loans that totalled more than the maximum allowed by the 
Program. We recommended that Industry Canada tighten its controls and 
examine the possibility of limiting to $250,000 the maximum guaranteed 
loans to related entities. 


5.43. The 1999 Act and Regulations define related borrowers more precisely 
and limit to $250,000 the financing to which related entities can have access. 
Furthermore, the Department now asks borrowers to declare on the loan 
registration form that they are not exceeding the allowable limit on financing. 
Also, the action plan for compliance audits of financial institutions includes 
audit procedures aimed at loans between persons with a non-arm’s-length 
relationship. 


The claims-processing procedure is effective 


5.44. The Department processed 3,748 claims in 2001-02. The average time 
it took to process a claim was 35 days from the receipt of all required 
information; over the last five years, this has fluctuated between 28 and 

45 days. These data reflect claims related to loans guaranteed under both the 
Small Business Loans Act and the Canada Small Business Financing Act. 


5.45 The Small Business Loans Administration Directorate uses a number 
of tools to ensure that staff assigned to claims processing follow uniform 
practices. It developed a procedures guide; and it documented its policies on 
the interpretation of the Act and Regulations and its decisions on new issues 
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in claims processing. It has also adopted an appeal mechanism to review 
decisions that lenders dispute. We noted that staff assigned to claims 
processing have the needed experience and expertise to process claims. 


5.46 The claims-processing procedure includes a number of steps; the most 
important is analyzing the information that the financial institution provided 
to justify the amount of its claim. In this step, the Department verifies 
whether the loan met the conditions of the Program (size of the business, 
activity, type of asset financed, interest rate, and terms of the loan) and 
whether the financial institution collected the collateral that secured the loan 
before submitting its claim. 


5.47. Our examination of a sample of claims processed in 2001-02 showed 
that the Department uses a valid procedure to ensure that claims are properly 
justified. 


The Department has reduced the interest paid to lenders 


5.48 A lender has up to three years to claim its losses on a defaulted loan. 
During that time, it must take all possible measures to collect on the loan 
before submitting a claim. The claim is to include the eligible proportion of 
the unpaid principal as well as interest on that amount. The $144 million in 
claims paid in 2001-02 included interest of $24 million and, on average, 
lenders submitted their claims 19 months after the borrower had defaulted on 
the loan. These data reflect claims related to loans made under both the Small 
Business Loans Act and the Canada Small Business Financing Act. 


5.49 In 1997 we recommended that the Department take appropriate steps 
to reduce the total amount it was paying in interest on claims submitted by 
financial institutions. 


5.50 The Regulations to the 1999 Act introduced a provision allowing a 
lender to submit an interim claim before having collected a borrower’s 
personal collateral, but after collecting on all other guarantees. In 2001-02, 
interim claims represented 23 percent of settled claims. 


5.51 The Regulations to the 1999 Act also shortened the period for which 
the government will pay interest to the lender. While lenders still have three 
years after a default to submit their claims, the Department no longer pays 
interest to the lender on that whole period. As it did before 1999, it pays for 
the first year a rate of interest equal to the rate at which the loan was issued, 
and for the second year it pays half that rate. As of 1999, however, it pays no 
interest for the third year. We estimate that in 2001—02 this measure 
contributed to a reduction of $800,000 in interest paid. 


5.52 Whether financial institutions can collect on loan security and submit 
their claims sooner depends on general economic trends as well as conditions 
in the local real estate and property trade-in markets. The realization of 
secured assets takes time, but it is essential; in 2001-02 the net amount 
realized in relation to claims settled in that year was $46 million. Current 
rules governing interest payments aim to strike a balance between the 


Report of the Auditor General of Canada—September 2002 Chapter 5 | i 


INDUSTRY CANADA—MANAGEMENT OF THE CANADA SMALL BUSINESS FINANCING PROGRAM 


Accountability to Parliament 


12 


* Chapter 5 


amount of interest paid and the due diligence exercised by financial 
institutions in realizing secured assets. 


Information provided to Parliament still needs improvement 


§.53 We noted in 1997 that Parliament was receiving mainly operational 
information on the Program and not the kind of information needed to assess 
its performance. The Department was providing no information on the extent 
to which it had achieved its cost recovery and incrementality objectives. We 
also questioned the reliability of its information on the job creation impact of 
the Program. 


5.54 In this follow-up, we reviewed the Department’s annual reports on the 
Program for the last five years. We noted that they now include better 
information on the achievement of the cost recovery objective. The 
Department has also clarified the meaning of the data it provides on the 
number of jobs created by the Program and has indicated some of the data 
limitations. We also saw more information on Program activities. 


5.55 Cost recovery. The annual report presents financial information on 
revenues and on claims expenditures. However, it provides no information on 
the Program’s expected financial performance. For example, the Department 
states that it will not achieve its cost recovery goal for loans guaranteed 
between 1995 and 1999. It gives no specific indication of the possibility of 
recovering its costs for loans guaranteed from 1999 on, because it believes it 
does not have enough data on claims for this period. 


5.56 Job creation. The Department now provides information on the 
expected number of jobs to be created, by industry sector and by business size. 
The information is presented with the caveat that it comes from borrowers’ 
projections of the number of jobs they expect to create. In 2002, Industry 
Canada commissioned a survey of firms that had obtained loans under the 
Program between January and July 2000. It asked them to confirm how many 
jobs they had created; most confirmed their initial estimates. The 
Department plans to repeat the survey at regular intervals. 


5.57 The Department still does not indicate clearly all the limitations of its 
data on job creation. The information does not account explicitly for 
potential job displacement due to the creation of a new business or the 
modernization of an existing business, or for the fact that a number of 
businesses that receive funding under the Program also obtain funding from 
other sources. 


5.58 Efficiency. The annual report contains little information on the 
administrative efficiency of the Program’s operations. For example, it does not 
provide data on the cost and the length of time required to process a claim, 
the number of claims denied, and the number of appeals on rejected claims 
and their results. 


5.59 The Department has informed us that it remains committed to 
improving the information it provides to Parliament in its annual report, 
particularly on the expected financial performance of the Program. 
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New issue of importance — ‘The decrease in the number of loans warrants further study 


5.60 The number of loans made through the Program has dropped 
significantly since 1998. To understand the reasons, the Department 
commissioned a study in the form of a survey of financial institutions; the 
study report was released in January 2002. The financial institutions see as 
the major reason for the decline the amount of administrative work involved 
in processing the loans and the Department’s growing tendency to deny 
claims. During our follow-up, we met with representatives of major financial 
institutions to obtain their views on the administration of the Program and to 
understand how they deliver it. They reiterated to us their concern about the 
administrative burden of the Program. 


5.61 Other reasons mentioned in the study were the lack of profit on these 
loans, a greater choice of financing options for borrowers, the possibilities for 
borrowers to get loans with better conditions, an increase in financing 
through internal means, and competition among lending institutions. The 
Department has not yet validated the conclusions of this study. 


§.62 The study provided the Department with interesting leads to explain 
the decline in loans under the Program. However, its scope and 
methodological limitations affected the depth of analysis and the reliability of 
conclusions. 


5.63 Recommendation. Industry Canada should 


* thoroughly examine the reasons for the decline in the number of loans 
and assess their impact on the rationale and the design of the Program, 
and 


* investigate the concerns of financial institutions about the 
administrative requirements of the Program. 


Industry Canada’s response. The Department has responded to the chapter 
overall; its response follows the Conclusion. 


Conclusion 


5.64 Industry Canada spent considerable time and effort in 1998 to review 
the Small Business Loans Act. The Act was subsequently replaced in 1999 by 
the Canada Small Business Financing Act. Some of the issues we had raised in 
our 1997 Report were addressed directly in the new Act with the inclusion of 
the due diligence principle in the legislation, clarification of the concept of 
related entities, and inclusion of measures to reduce interest paid on claims 
submitted by lenders. 


5.65 The Department has taken measures to improve the management of 
the Program. It developed an evaluation framework and has begun to collect 
data to measure the Program’s performance. It has yet to define criteria for 
assessing the Program’s success. We are still concerned about the 
Department’s ability to achieve the cost recovery objective of the Program. 
It has improved its capacity to monitor the Program’s financial performance 
but has yet to enhance its claims forecasting model. 
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5.66 The Department has also made progress in its accountability to 
Parliament. It now provides information on the Program’s revenues and 
expenditures but not on its expected financial performance. The Department 
provides little information on the limitations of its job creation data and the 
efficiency of its operations. 


5.67 The decline in the number of loans under the Program is a new issue 
the Department needs to monitor. It needs to understand the reasons for this 
drop and the impact on the rationale and design of the Program. 


Industry Canada’s response and comments. As recognized by the Auditor 
General in the chapter’s conclusion, Industry Canada has made considerable 
progress with respect to issues raised in the Auditor General’s 1997 Report. In 
addition, the Department expects that further progress will be achieved in the 
fall of 2002 with the implementation of its audit plan and a new results-based 
management and accountability framework, which will further define 
performance indicators for the Program. 


Industry Canada acknowledges that continuous monitoring of the Program is 
essential to achieving its objectives of incrementality and cost recovery. 
Concerning cost recovery, Industry Canada forecast in its 2000-01 Annual 
Report that over the life of the loans made in 1995 to 1999, disbursements on 
claims will exceed revenues received. The development of an improved 
forecasting model is a high priority and has been since the introduction of the 
new Act. Having collected program data for three years, consistent with the 
new Act, Industry Canada is now positioned to adjust its forecasting model. 
In addition, Industry Canada will be reviewing these data to determine 
whether to recommend any changes to the Program to meet the cost recovery 
objective. 


Finally, Industry Canada will, as recommended, continue to carefully monitor 
changes in the level of use of the Program. The administrative requirements 
will also be monitored in order to achieve the appropriate balance between 
the informational and regulatory requirements necessary to meet our 
performance measurement and other commitments on the one hand, and the 
burden on lenders and borrowers in responding to these conditions on the 
other. 
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About the Follow-Up 


Objective 


The overall objective of the follow-up audit was to assess the progress made by Industry Canada in addressing the 
issues raised in our December 1997 Report. We wanted to determine whether the Department now has the systems 
and practices in its Program to manage it efficiently, cost-effectively, and in compliance with legislative authorities. 


Scope 


The follow-up covered Industry Canada’s management of the Canada Small Business Financing Program and its 
administration of the Small Business Loans Act and Canada Small Business Financing Act, which govern the Program. 
(The follow-up excluded the capital leasing pilot project launched by the Department in April 2002 to make capital 
leasing more accessible to SMEs.) We examined the responsibilities and activities of Industry Canada, in particular 
the Small Business Loans Administration directorate and the Small Business Policy Branch, and those of lenders. 


Criteria 


Our examination was based on the following criteria: 


¢ The objectives and the expected results of the program related to small business loan guarantees should be 
clearly stated. 


* The Department should have management practices and controls that would ensure that small business loan 
guarantees are processed efficiently and cost-effectively, that the risks of the program are minimized, and that 
the objectives of the program are achieved. 


¢ Parliament should be provided with relevant, reliable, and understandable information on the economy, 
efficiency, and effectiveness of the program. 


Approach 


We interviewed Industry Canada employees and representatives of financial institutions. We reviewed the studies 
and reports produced by and for Industry Canada and examined the Department’s systems and practices. We also 
conducted analyses based on the Department’s data. 


Ratings 
We assessed the action of departments/agencies against our original audit recommendations (see Key Message at the 
beginning of the Chapter). We used the following ratings: 

* Completed. Corrective action has been fully implemented. 

* Satisfactory progress. Progress is being made at a satisfactory pace. 

* Limited progress. Some progress is being made, but the pace or scope is not satisfactory. 


- No progress. No evidence of progress although the department or agency accepted the recommendation from 
the original audit. 


* Rejected. The department or agency did not accept the recommendation from the original audit. 


* Unknown. Status of progress is unknown or information is not available. 
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Making a difference—for 125 years 
I’m pleased to present my second Status Report to the House of Commons. 


When I was named Auditor General in May 2001, I decided to keep the 
slogan—Making a Difference—that had been adopted by my predecessor. 
Though many other organizations can and do make the same claim, I think it 
sums up the mandate and role of the Office of the Auditor General in a 
simple and straightforward way. 


Providing independent information to parliamentarians in my regular reports 
to the House of Commons—along with assurance on information provided by 
government—is an important step in the process of making a difference for 
Canadians. The information generated through our audits and studies helps 
parliamentarians better scrutinize government programs and services and 
determine whether or not Canadians have received value for their tax 
money—key to Parliament’s role as watchdog over the public purse. The 
information also helps them evaluate the progress being made in government- 
wide management reforms. 


Parliamentarians have the important task of making sure that steps are taken 
to correct any problems we identify. My colleagues and I frequently appear 
before parliamentary committees, along with representatives of the 
departments we have audited, to answer questions about what our audits 
found. The Standing Committee on Public Accounts often asks departments 
to produce action plans for correcting problems, and it holds them to those 
plans; occasionally other standing committees do this too. 


This Status Report also plays a role in making a difference: it closes the loop 
in the auditing process by reporting back on action the government has taken 
in response to our audit findings and recommendations. We introduced this 
report in September 2002 to better meet the needs of parliamentarians, 
particularly the members of the Standing Committee on Public Accounts. 


To discharge their responsibilities well, our elected representatives need 
answers to key questions: What steps has the government taken to make 
needed improvements? How well are departments and agencies progressing in 


those efforts? 


We designed the Status Report to provide clear and focussed answers to these 
questions, based on a rigorous follow-up on our recommendations. It allows 
us to answer some of our own questions as well: Does our work promote 
positive change? Does it in fact make a difference for Canadians? 


I think this Report provides solid evidence that government does act in 
response to our recommendations; we provide many examples of efforts by 
departments and agencies to make changes that will have lasting benefits. 
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Overall, I am pleased to say that the results are positive. Of the six chapters 
that follow, four report satisfactory progress, while two—Rating 
Departmental Performance Reports and Reform of Classification and Job 
Evaluation in the Federal Public Service—report unsatisfactory progress. 


Managing the Risks of Non-Compliance for Customs 


Overall, we are satisfied with the Canada Customs and Revenue Agency’s 
progress in responding to the 10 recommendations we made in two previous 
audits of the Customs program, given the difficulty of the recommendations 
and the amount of time that has elapsed since we made them. Our 
recommendations were aimed at improving risk management in the Customs 
program. In April 2000 our focus was on risks related to travellers entering 
Canada; in December 2001 it was on risks related to commercial shipments. 
When we returned recently to do our follow-up, we found that the Agency 
had made satisfactory progress in implementing roughly half our 


recommendations. 


In the aftermath of September 11, 2001, the Agency moved quickly to 
increase the screening of travellers and commercial shipments to detect 
potential terrorist threats. But it still faces many challenges as it seeks to 
manage the risks at Canada’s borders. 


We are pleased to see that advance passenger information is now collected 
from most airlines to help the Agency target high-risk air travellers. A new 
system now used at airports allows for better screening of travellers by 
customs officers. The Agency has also improved the targeting of in-transit 
marine containers by setting up joint targeting units with United States 
customs officials. 


It has begun to implement its Compliance Improvement Plan—the overall 
compliance strategy for the Customs program and a key element of managing 
risk intelligently. To better clarify roles and responsibilities, the Agency has 
developed a plan to complete or update memoranda of understanding with 
organizations on whose behalf it works at the border, and it has already signed 
five. 

The Agency has put an action plan in place to strengthen its compliance 
verification regime, which checks whether importers are complying with 
trade laws and regulations. It is also developing an ambitious training program 
and long-term learning plan for its compliance verification officers. 


We are not satisfied with the progress made on our other recommendations. 
The Agency needs to move faster on collecting the information it needs to tell 
Parliament whether its risk management strategy is working and to support its 
assessment of how well the Customs program is meeting its goals. It must also 
step up efforts to analyze the results of random examinations; this would assist 
in identifying and targeting high-risk travellers and commercial shipments. 


A lack of good records on training hampers the Agency’s ability to determine 
whether its customs officers have received the training they need. Customs 
officers continue to express concerns about their training, even though the 
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Agency offers many courses for them. And while it has started to collect 
information from other government entities about the risks that arise from 
responsibilities undertaken on their behalf, the Agency needs to do much 
more to incorporate this information in its risk assessments. 


The Economic Component of the Canadian Immigration Program 


In April 2000 we reported serious problems in the management and delivery 
of the Canadian Immigration Program’s economic component, which applies 
mainly to skilled workers and business immigrants. We noted that these 
problems hindered Canada’s ability to protect the integrity of the Immigration 
Program and to make the most of the economic and social benefits of 
immigration. 


Our follow-up audit found that Citizenship and Immigration Canada has 
taken our 2000 report and its recommendations seriously, as did the Standing 
Committee on Citizenship and Immigration when it considered the new 
Immigration and Refugee Protection Act. Overall, we are satisfied with the 
progress the Department has made. 


In 2002 the Department implemented the new Act and its regulations—the 

first major overhaul of immigration legislation in some 25 years. Discussions 

with visa officers and other overseas staff indicate that the Department did a 
good job of preparing employees for the introduction of the new legislation by 
providing training and information. 


Many of the issues raised in our 2000 audit have been addressed by the new 
Act and regulations. For example: 


¢ Selection criteria for skilled workers and business immigrants have been 
amended. 


¢ “Excessive demand” for health and social services has been defined. 


¢ The Department can now apply for non-disclosure of information 
during admissibility hearings, immigration appeals, and judicial reviews. 


Submission of fraudulent documents and misrepresentation are an ongoing 
threat to the integrity of the Immigration Program. To address this, in 2002 
the Department created a new Intelligence Branch and signed new 
memoranda of understanding with the Royal Canadian Mounted Police and 
the Canadian Security Intelligence Service. A new tool, the Modern War 
Crimes System, is available to help visa officers identify war criminals. In 
addition, a new Permanent Resident Card has been developed. 


We are not satisfied with the Department’s progress in addressing concerns 
we raised in three specific areas: medical surveillance of immigrants, quality 
assurance for selection decisions, and the quality of reporting to Parliament. 


The medical surveillance of immigrants needs greater attention to ensure that 
public health authorities are notified promptly of individuals requiring 
surveillance. The Department does not know what percentage of immigrants 
report to public health authorities and within what timeframe. 
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Although the Department has been developing a method for monitoring the 
quality of selection decisions for immigrants who fall within the economic 
component, we note that a department-wide quality assurance framework is 
not yet in place. This framework is essential to ensure the consistency, 
fairness, and integrity of the selection process. 


Finally, we urge the Department to provide better information to Parliament 
on the economic component of the Immigration Program. Reports remain 
focussed on the numbers of immigrants admitted, although the Department 
does have other readily available information that could be useful to 
Parliament, such as the number of applications on hand and processing times. 
Currently, the number of skilled workers reported as entering Canada 
includes both the principal applicants and their dependants, which may lead 
people to overestimate the number of skilled workers admitted. 


One new issue emerged in the course of this follow-up audit—the medical 
surveillance of refugee claimants. 


Refugee claimants are required to undergo a medical assessment after they 
arrive in Canada. However, when it is determined that they require medical 
surveillance for inactive tuberculosis, the Department does not notify them or 
the public health authorities in the provinces and territories unless the 
claimant applies for a temporary work or study permit. Refugee claimants are 
not under medical surveillance while their claims are being processed by the 
Immigration and Refugee Board. This process took 10.4 months on average in 
2001-02. If the Board denies refugee status, the public authorities are not 
notified and the failed claimant can remain in Canada for several months or 
years. 


Correctional Service Canada—Reintegration of Male Offenders 


In the past 10 years, five of our audits have reviewed all major aspects of the 
process of reintegrating male offenders into the community—a vital part of 
the mandate of Correctional Service Canada. 


Overall, we are satisfied with the progress the Service has made in addressing 
many of our recommendations. It has taken positive steps to strengthen 
compliance with standards for supervising offenders in the community, 
improve the quality of reports to the National Parole Board for parole 
decisions, institute more timely case management, and shorten the time it 
takes to acquire critical documents on offenders. 


It has been hard to make progress in many of these areas because the large 
number of parole officers who contribute to offender reintegration are widely 
dispersed across Canada. The commitment and leadership of senior 
management has been the key to overcoming these difficulties and achieving 
results. 


We are not satisfied with the progress the Service has made in some other 
areas. There are still not enough employment programs or offender 
rehabilitation programs in the community. Senior correctional officers have 
not improved their performance of case management duties, and the Service 
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has not removed all delays within its control that affect the timely 
preparation of offenders for conditional release. 


We recognize that substantive progress in many of these areas requires a 
fundamental change in the Service's operational culture as well as improved 
management practices, and thus will demand a sustained effort. 


The follow-up also identified some additional related issues that need 
attention. We recommend that the Service take steps to 


* further test the tools it uses in the initial assessment of offenders, to 
ensure that different raters using the tools would arrive at consistent 
results (inter-rater reliability) ; 


* provide better initial and ongoing training for parole officers; 
* examine the extent and impact of turnover among parole officers; and 


* ensure that offenders’ correctional plans are completed on time. 


These are all fundamental ingredients in the sound reintegration of offenders. 


Post-Secondary Recruitment Program of the Federal Public Service 


The Public Service Commission has redesigned its Post-Secondary 
Recruitment program to better meet the needs of hiring managers and 
graduating students looking for work in the public service. In so doing, it has 
substantially implemented most of the recommendations we made in our 
December 2000 Report and has made recruiting faster and more flexible. 
Although some managers are still unaware of the recent changes, we are 
satisfied with the Commission’s progress in marketing the program to 
departments. 


Over the last three years, departments have increased the number of jobs 
they have filled using the program. However, the increase has been small 
compared with the increase in the total number of people hired into the 
public service during the same period. Despite the program’s successful 
redesign, departments are not yet taking full advantage of it. Our follow-up 
found that while several departments have made significant progress, some— 
have made only limited progress in human 


and the government as a whole 
resources planning and in establishing recruitment targets. 


In December 2000, we recommended that the government develop a results- 
oriented recruitment strategy that would identify post-secondary recruitment 
targets to address workforce renewal challenges for the years ahead. The 
government still lacks a complete picture of both its recruitment needs and 
the educated and skilled people who are entering the public service through 
its various recruitment routes. Many managers continue to adopt the 
expediency of short-term hiring, despite the opportunity to recruit through a 


fast and flexible program. 


Departments need to balance hiring to fill immediate vacancies and hiring 
strategically, with a view to ensuring that the government will have qualified 
people to deliver its programs and services in the future. 
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Rating Departmental Performance Reports 


Every fall, federal departments and agencies are required to submit 
performance reports to Parliament that outline the results they have achieved 
in the previous fiscal year. Good reporting about results serves two important 
purposes: it enables parliamentarians to hold ministers to account for the 
funds their departments spend and it informs Canadians about the value they 
are getting for their tax dollars. Thus, good performance reporting is essential 
to the government’s accountability to Parliament and to Canadians. 


The first chapter of this report takes a look at these departmental 
performance reports and assesses how useful they are in providing credible 
information. Although our first examination of the government's reporting 
system in 1997 showed that the basic framework was sound, when we took 
another look at performance reports in 2000 we concluded that departments 
and agencies were too slow in improving the quality of their reports. 


In 2002, we presented a model for rating departmental performance reports 
that outlines criteria of good reporting. In this report, we use that model to 
rate nine performance reports. 


The overall result is still disappointing. Although we identify some promising 
practices, we again find that departments need to move more swiftly to 
improve their performance reports. Reports tend to share certain weaknesses: 
the results reported are not always credible and balanced; there is scant 
evidence that departments use performance information to make decisions; 
and there is too little reporting on important horizontal issues, such as 
security, that span the mandates of several government departments. 


We will continue to update our rating model and use it both to encourage 
improvements in the quality of departmental performance reports and to 
identify promising practices and key challenges. 


Reform of Classification and Job Evaluation in the Federal Public Service 


Since 1995, we have been auditing the government’s initiative to reform its 
classification and job evaluation systems through the development of the 
Universal Classification Standard. Normally we would follow up on previous 
recommendations but since the government has significantly changed its 
approach to classification reform, we focussed on its progress from 2000 to 
2002 and the challenges that remain. 


Overall, we are not satisfied with the progress made in reforming the 
government's outmoded and problem-ridden classification system, and we are 
concerned about the significant challenges the Secretariat faces in its new 
approach to achieving its objectives. 


The system the federal government uses to classify and compensate positions 
is now almost 40 years old. Skills that were highly valued when this system 
was created have become obsolete, and changes in technology have 
transformed the way government works, creating new occupations in the 
process. The balance has shifted away from operational and administrative 
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support categories to policy development, program management, and 
professional and technical specialties such as law, science and economics. 


It is clear that the old system, which is still in use today, has many problems: it 
is costly, cumbersome, time-consuming and complex. The government has 
been trying since 1991 to reform the classification and job evaluation system 
in the federal public service in an effort to resolve long-standing problems. 


However, in May 2002 the government declared that its second attempt at 
reform (1995 to 2001), the Universal Classification Standard, could not be 
implemented government-wide because it could not develop a single 
compensation structure for all employees in the public service that would be 
workable. This planned universal approach has now been abandoned, despite 
a large investment of time and effort by tens of thousands of employees and 
an estimated investment of about $200 million in incremental costs 


between 1998 and 2001. 


The Secretariat announced in May 2002 that the government would begin a 
new Classification program of step-by-step, tailored reforms. Like the old 
system, it will use separate classification standards for each occupational 
group. The Secretariat is committed to using some of the work completed 
under the Universal Classification Standard project as a basis for developing 
the new group-specific standards, thereby recouping part of the investment 
made in developing the Standard. Our follow-up audit found the Treasury 
Board Secretariat still at the planning stage of the new approach. 


The Secretariat will have to manage a number of challenges to succeed in this 
third attempt. It still has not defined a clear vision of what classification 
reform will entail and has not articulated an overall plan and timetable for 
completing the incremental reform. A step-by-step approach represents 
significant risks when the overall results are not defined. Our audits of other, 
similarly ambitious government-wide reforms have found that consistent 
leadership, clear expected results, sufficiently detailed plans with timelines 
and milestones, and active monitoring are essential to success. 


In addition, the Secretariat has not finalized its compensation policy and 
objectives. It has addressed neither competing forces such as internal equity, 
labour market demands, and affordability nor the continuing need to satisfy 
the requirements of the Canadian Human Rights Act. 


Finally, in focussing its efforts on the Universal Classification Standard 
project, the Secretariat did not exercise control over the classification of 
positions under the old, still existing classification system—so no one knows 
to what extent positions are misclassified. We view the state of job 


classification and compensation with serious concern. 
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Conclusion 


We recognize the enormous challenge the government faces in delivering 
large and complex programs and services. Our audits do find shortcomings— 
that is the nature of auditing—but we try to be constructive in our criticisms 
and suggest solutions to the problems we identify. When we make 
recommendations for change, it is with the utmost respect for federal 
institutions and for the men and women who dedicate themselves to public 
service. As legislative auditors, we share an important responsibility with 
government: safeguarding the public purse—we, through our audits and 
these follow-up audits; and government, through its actions to respond to our 
recommendations and ensure sound management of public resources. 


In conclusion, we are satisfied with the progress we found in four of our six 
follow-up audits this year. | hope this Status Report reassures those who 
wonder whether anything is ever done to fix problems identified in our audits. 
Clearly, the answer is yes, thanks to both Parliament and government 
departments. I congratulate the members of the public service who view our 
recommendations as opportunities to manage better. Canadians reap the 
benefits and the results, whether in the form of better accountability, 
improved programs, increased efficiency, lower costs, or better service. 


Since the first independent Auditor General was appointed 125 years ago, the 
Office has worked diligently to serve Parliament. By consistently identifying 
ways to improve the government’s performance, and by reporting back on 
concrete action that has been taken, the Office of the Auditor General strives 
to continue its valued tradition of making a difference for Canadians. 
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All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 
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Rating Departmental Performance 
Reports 


Main Points 


1.1 Performance reporting is more important than ever for 
parliamentarians and Canadians. As events unfold and debates take place on 
health, the environment, and security, Canadians want to know the value 
they are getting for their taxes and that their elected representatives can hold 
ministers to account for the money their departments spend. Departmental 
performance reports are a key tool for ensuring effective accountability. 


1.2 = In 2000 an audit of departmental performance reports concluded that 
the progress federal departments and agencies have made to improve the 
quality of their performance reporting to Parliament was too slow. In 2002 we 
presented a model to rate departmental performance reports. 


1.3. In this audit, we rated nine departmental performance reports tabled in 
the fall of 2002 against this model. Through this exercise we identified some 
promising practices, as well as many challenges that need to be addressed. 
Overall, we found that progress in performance reporting remains too slow for 
these nine departments. 


14 We found that performance reports provide a good overview of each 
department’s organizational context and planned strategic outcomes. 
However, performance expectations are not always clear and concrete, 
information does not necessarily focus on program results, and reported 
results are not always credible and balanced. As well, the reports provide little 
evidence that performance information is used to make decisions. We also 
looked at one horizontal issue and found that the reports give only limited 
recognition to the importance of reporting on such issues. 


15 We will continue to update and use our rating model to encourage 
improvement in the quality of departmental performance reports, as well as to 
identify promising practices and key challenges. 


The government has responded. The government welcomes the efforts of 
the Auditor General to assess departmental performance reports. Its 


comments are included at the end of this chapter. 
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Introduction 


Performance information continues to be important 


16 The Office of the Auditor General believes that good performance 
reporting is fundamental to effective governance and accountability to 
Parliament. 


1.7 Performance reporting is important for Canadians who want to know 
the value they are getting for their tax dollars. It enables them to understand 
the difference a department is making for them. Parliamentarians also need 
timely access to good quality performance information about government 
activities, so that they can scrutinize programs and services more effectively. 


18 Every fall, over 80 federal departments and agencies submit 
performance reports to Parliament that outline their accomplishments to the 
end of the last fiscal year as established in previous reports on plans and 
priorities. 


1.9 In 1997 we examined the state of the federal government’s reporting 
system following the implementation of the Improved Reporting to 
Parliament Project that was initiated two years earlier. We found that the 
government had made a good start and that the basic reporting framework 
was sound. However, when we conducted a follow-up audit in 2000, we found 
that the pace at which departments were making the needed improvements 
to their performance reports was too slow. 


1.10 In 2001 the Eighth Report of the Standing Committee on Public 
Accounts stressed the importance of good performance reporting. The 
Committee also recommended that we “conduct random audits of the 
information contained in the performance reports of departments and 
agencies in order to verify, among other things, that the information 
contained in these reports is a fair representation of accomplishments against 


goals and objectives.” 


1.11 In response to this request, we developed a model to rate performance 
reports, which was released in 2002. This model outlines the criteria 
necessary to achieve good reporting. We are still updating our model and will 
adjust it as we use it to rate performance reports so that it will continue to 
reflect current best practices in reporting. 


1.12 The Treasury Board Secretariat stressed the importance of good 
performance reporting in Results for Canadians: A Management Framework for 
the Government of Canada (2000) which stated that “Strengthening 
accountability to Parliament and to citizens is an integral part of the 
management board’s change agenda.” In 2002 the President of the Treasury 
Board released a report entitled Canada’s Performance 2002 that summarizes 
Canada’s overall performance on 19 societal indicators. A similar report was 


released in 2001. 
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1.13 The Treasury Board Secretariat and the Office of the Auditor General 
plan to release the Managing for Results Self Assessment Tool in 2003. This 
tool will describe the transition path toward effectively managing for results. 
Many of the tool’s concepts are similar to those of performance reporting as 

both approaches require credible and balanced results information. 


114 The Treasury Board Secretariat's guidelines for preparing departmental 
performance reports outline performance reporting principles and provide 
advice about how to interpret and apply them. Even though there is no one- 
to-one relationship between the model and the Treasury Board Secretariat's 
guidelines for performance reporting, the criteria are broadly consistent with 
the principles in the guidelines. The model has received the general support 
of the Secretariat. The principles in the CCAF’s (formerly the Canadian 
Comprehensive Auditing Foundation) 2002 Reporting Principles— Taking 
Public Performance Reporting to a New Level are also consistent with our 
criteria. 


A model for rating departmental performance reports 


1.15 Given the importance of performance reporting, we continue to work 
to improve our model for rating departmental performance reports. This 
model rates reports according to five criteria. A performance report that fully 
meets all five criteria would demonstrate the attributes of excellent reporting. 
The five criteria, each of which include several sub- criteria, are as follows: 


* organizational context and strategic outcomes are clear, 
* performance expectations are clear and concrete, 

* key results are reported against expectations, 

* performance information is credible and balanced, and 


* use of performance information is demonstrated. 


1.16 ‘Together, the criteria represent expectations for a credible performance 
story about the difference a department can make for Canadians 

(Exhibit 1.1). The first three criteria reflect what has been accomplished; the 
other two indicate the quality and use of the performance information. In 
particular, the fourth criterion expects the department to show what it does to 
ensure the quality of its performance information. However, our model is not 
designed to provide assurance that the information in a performance report is 
accurate because it does not include an audit of performance information. As 
well, if a report omits a key piece of performance information, it would not 
likely be taken into account in the rating. 


1.17. Our model provides five achievement levels. The fifth level represents 
the attributes of excellence in performance reporting. By meeting each 
criterion of the model at progressively higher levels, performance reports 
demonstrate that departments have increasingly mastered these attributes. 


4 Chapter 1 
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Exhibit 1.1 Model for rating performance reports—Overview 


Criterion 


Organizational context and 
strategic outcomes are clear 


Level 1 Level 2 Level 3 Level 4 Level 5 
(basic) (fair) (good) (very good) (excellent) 


| 
| | | 


Planned strategic outcomes at each level are increasingly aligned in logical sequence with the 
department's operating environment, its legislated mandate, mission, and relevant risks. They 
indicate how the department and its key partners will contribute to the strategic outcomes. 


| | | 


Performance expectations are 
clear and concrete 


| 


Statements of performance expectations are increasingly expressed as outputs and outcomes 
with a direction, an amount of change, a timeframe, and a strategy for achieving planned targets. 


| | | 


Key results are reported 
against expectations 


| | 
| | | | 
Key results are increasingly expressed as outputs and outcomes that contribute to each of the 


planned strategic outcomes with results that are aligned with performance expectations. They 
are increasingly accompanied by challenges, attribution, and resource information. 


| | | | 


Performance information is 
credible and balanced 


| | 


Performance information is increasingly supported by reliable sources and information on 
data quality. The performance information presented includes a balance between 
successes and shortcomings. 


| | | | 


Use of performance 
information is demonstrated 


| 
| | | | 
Performance information is increasingly used to manage and improve future performance. 


| Sle | | 


Focus of the audit 
1.18 The objective of this audit is to rate several departmental performance 
reports by 


* determining the overall quality of these departmental performance 
reports, which provide Parliament and Canadians with information 
about departmental performance; and 

* determining if horizontal issues (paragraph 1.57) are properly identified 


in these performance reports, using a shared outcome as an example. 


1.19 We selected the performance reports of the majority of departments 
and agencies that have mandates that relate to the security of Canadians. 


These are 
¢ Citizenship and Immigration Canada, 
* Correctional Service Canada, 
« the Department of Justice Canada, 


¢ the Immigration and Refugee Board, 
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« National Defence, 

* the National Parole Board, 

« the Royal Canadian Mounted Police (RCMP), 
« the Solicitor General Canada, and 


¢ Transport Canada. 


These organizations represent $19.3 billion in expenditures, or 12 percent of 
the government’s total expenditures for fiscal year 2001-02. 


1.20 More details about this audit’s objectives, scope and approach are 
provided in About the Audit at the end of this chapter. 


Observations and Recommendations 


1.21. Our audit found that the reports are not always as clear and useful as 
they need to be (Exhibit 1.2). Overall, the reports achieved a fairly good 
rating on the first criterion. They almost reached, or reached, a fair rating on 
the second, third and fourth criteria. However, they barely exceeded the basic 
level on the fifth criterion. These results show that the performance reports 
we rated have a long way to go before they reach the highest levels of the 
model. Several previous audits also expressed significant concern about the 
overall quality of reporting. Yet, based on our rating of these nine reports, this 
audit shows that performance reporting continues to face many challenges. 
Nonetheless, we have identified some promising practices in the nine reports 
that we rated, which illustrate how performance reports can be improved. 
These practices show how it is possible to improve reporting to Parliament. 


Exhibit 1.2 Summary results for the nine reports rated 


Organizational context and strategic outcomes 
are clear 


Performance expectations are clear and concrete |__| 
Key results are reported against expectations 


Performance information is credible and balanced 


Use of performance information is demonstrated 


Levels 2 3 4 5 
Basic Fair Good Very good Excellent 


Organizational context and strategic 1.22 The first criterion requires that a department's organizational context 


outcomes and strategic outcomes are clear. It assesses whether the report clearly states a 
department’s business lines and whether these are reflected in its mandate 
and mission. It also determines if there is a logical sequence and link between 
its business lines and its key results, which include its planned strategic 
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outcomes. These factors should be aligned with those that influence an 
organization’s operating environment. This criterion also asks that key 
external partners be identified and that a credible description of the risks the 
department faces be provided. 


1.23 The first criterion assumes that public sector organizations do not exist 
in a vacuum. That is, they are subject to many internal and external 
opportunities and pressures, and they have to solve increasingly complex 
issues in order to fulfil their mandate. Therefore, a department’s performance 
story is only meaningful when it is placed in this broader context. 


The reports generally provided a good overview of the organizational context and 
strategic outcomes 


1.24 All but one of the reports achieved a rating of fair to very good on the 
first criterion (Exhibit 1.3). This is consistent with what we found in our 
2002 Report when we demonstrated how our model could be used. 


Exhibit 1.3 Organizational context and strategic outcomes are clear—Results for the nine reports rated 


Citizenship and Immigration Canada 
Correctional Service Canada 
Department of Justice 

Immigration and Refugee Board 


National Defence 


National Parole Board 
RCMP 


Solicitor General Canada 


Transport Canada 


Levels 1 2 3 4 5 
Basic Fair Good Very good Excellent 


Promising practices in reporting organizational context and strategic outcomes 


1.25 The reports provide good overviews of their planned strategic 
outcomes. Transport Canada, National Defence, and Citizenship and 
Immigration Canada provide Web links in their reports so that readers can 
get more background information. By providing more comprehensive 
information about the department and its organizational context through 
these Web links, this approach helps to keep the reports comparatively short 
and focussed on performance. 


1.26 The reports also provide good information about how the department’s 
business lines contribute to planned strategic outcomes, and how they are 
aligned with their missions and mandates, as well as federal government 
priorities. Transport Canada and the Solicitor General Canada did well in this 
area. Transport Canada’s report includes a graphic that clearly shows how it 
links its strategic objectives with its business lines, resources, and broad 
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government themes. It also shows how the department’s activities and 
outputs contribute to its strategic objectives. 


Challenges to reporting on organizational context and strategic outcomes 


1.27. Although the reports did relatively well with respect to the first 
criterion, there is still room for improvement. An important aspect of 
organizational context is the key partnerships through which departments 
deliver some of their programs. In most cases, only a list of partners is 
provided. There is no information about which department has the lead, or 
how the activities and outputs contributed to a shared outcome. We address 
this issue again when we discuss the reporting of horizontal issues. 


1.28 According to the second criterion, performance expectations should be 
clear and concrete. That is, departments should clearly state what they will 
do for Canadians. This criterion focusses on performance expectations and on 
how they relate to the organization’s strategic and operational plans. 
Performance expectations should be clearly identified and aligned with 
federal government priorities. They should also be expressed as outputs, or 
outcomes at higher levels in the model. They should include the direction of 
planned changes and the timeframe in which those changes will be made. 
The report should state, for example, how much outcomes will be increased, 
maintained, or decreased as a result of the department’s programs. 


1.29. The second criterion also captures how the organization plans to 
achieve its performance goals. While most of this information is included in 
the corresponding report on plans and priorities, the performance report 
should contain enough information to enable readers to understand how the 
department’s strategies and activities will achieve the expected results. 


Performance expectations are not always clear and concrete 


1.30 With the exception of two departments, the reports did not rate well 
on the second criterion. Overall, we found that performance expectations are 
not clear and concrete, and that they are not aligned with the organization’s 
strategic and operational plans (Exhibit 1.4). 


Exhibit 1.4 Performance expectations are clear and concrete—Results for the nine reports rated 


Citizenship and Immigration Canada 
Correctional Service Canada 
Department of Justice 

Immigration and Refugee Board 
National Defence 

National Parole Board 

RCMP 


Solicitor General Canada 


Transport Canada 


Levels i} 2 3 4 5 
Basic Fair Good Very good _ Excellent 
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Promising practices in reporting performance expectations 


1.31. The RCMP report provides an executive summary chart that assesses 
its performance against the problem being addressed. However, because the 
chart does not include the organization's specific performance expectations, it 
is not as informative as it could be. 


132. Throughout the Citizenship and Immigration Canada report, clear and 
concrete statements are made about how the department’s performance 
expectations could affect Canadians. For example, it describes how achieving 
certain immigration levels will offer economic and social benefits for Canada. 
It explains how immigrants play an important role economically by building 
human capital and filling labour market gaps. According to the report, 
Canadian society also benefits from the cultures, traditions, and knowledge 
that immigrants offer, and from the strong communities and social stability 
that reunited families provide. The report states that attaining this 
performance expectation is to be demonstrated by “achieving planned 
emigration levels of 200,000 to 225,000 newcomers to Canada for 2001.” 
This performance expectation states clearly the direction, amount, and 
timeframe of the planned change. 


1.33. The Immigration and Refugee Board performance report discusses 
strategies for achieving planned levels of activities and outputs and how to 
deal with high volumes of outstanding immigration claims. For example, the 
report states that resources will be used to develop streamlined processes to 
better manage the record number of outstanding refugee claims. 


Challenges to reporting performance expectations 


1.34 Reports on plans and priorities are intended to establish performance 
expectations and outline the general direction a minister wants a department 
to take during the Estimates year and the two fiscal years that follow. Because 
performance reports are designed to provide information on results that were 
achieved, they are compared with the relevant reports on plans and priorities. 
Reports on plans and priorities set performance targets while performance 
reports state the extent to which they were met, and provide explanations for 
the level of performance achieved. 


135 We found that, in many cases, performance reports were not consistent 
with departments’ reports on plans and priorities. In fact, in seven of the nine 
reports that we rated, the performance expectations in the planning reports 
were, to varying degrees, different from those in the performance reports. In 
some cases, performance expectations were added, consolidated, modified, or 
deleted, and the differences were not explained. As a result, accountability 
for previous commitments is not clear. For example, the Solicitor General 
Canada’s report is inconsistent with the Department’s report on plans and 


priorities. 


1.36 We also found cases where targets are vague. For example, the RCMP 
report states that a reduction in organized crime would be demonstrated by 
long-term investigations that indicate results that are superior to those in the 
previous year. Another practice is to provide targets that are activity-based 
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Performance outcomes 


An outcome is an external consequence that 
can be attributed to an organization, program, 
etc. and that is considered to be significant in 
relation to its commitments. Outcomes may be 
described as immediate, intermediate, final/end, 
direct or indirect, intended or unintended. 


An output is a direct product or service that is 
produced through program activities and 
delivered to a target group or population. 


(Treasury Board Secretariat) 


rather than ones that focus on outcomes. For example, the Department of 
Justice report states that a fairer, more effective youth justice system will be 
measured by the youth justice legislation. 


1.37. The third criterion requires that key results be reported against 
performance expectations and that departmental contributions be addressed. 
It focusses on results that are expressed as outputs, or outcomes, at the higher 
levels in the model, and whether or not the department achieved what it said 
it would achieve. The elements of this criterion include aligning outputs and 
outcomes with performance expectations, addressing the challenges of 
achieving expected results, identifying who contributed to the performance 
outcomes, determining the level of resources required, and explaining how 
results are interpreted. 


1.38 This complex criterion is at the heart of good performance reporting. 
Unlike the first two criteria, which set the stage for performance reporting, 
this criterion addresses the key attributes of the performance information that 


is reported. 


Outcomes are not widely reported 


1.39. The ratings for the third criterion ranged from basic to good. In 
particular, the amount of results information reported by departments varies a 
ereat deal. Overall, we consider the results to be very mixed (Exhibit 1.5). 
Five reports provide information on the level of resources needed to achieve 
the strategic outcomes but none had information on resources for key 
outputs. As well, they pay little attention to who is responsible for achieving 
these outcomes. 


Exhibit 1.5 Key results are reported against expectations—Results for the nine reports rated 


Citizenship and Immigration Canada 
Correctional Service Canada 
Department of Justice 

Immigration and Refugee Board 
National Defence 

National Parole Board 

RCMP 


Solicitor General Canada 


Transport Canada 


Levels 1 Z 3 4 5 
Basic Fair Good Very good Excellent 


Promising practices in reporting outcomes 


1.40 The Immigration and Refugee Board report provides a lot of detailed 
information about its operational outputs. For example, the Board’s report 
provides data about several referred and finalized refugee claims, specifically, 
about the average processing time and the cost of processing each claim. 
Despite the fact that it does not report performance expectations, 
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Correctional Service Canada does report some quantified information for a 
number of key results such as re-offending rates during supervision. 


1.41 Many of the reports provide Web links to their evaluations and internal 
audits. For example, the RCMP report includes a section that conveniently 
summarizes its evaluations and internal audits. While the Citizenship and 
Immigration Canada report indicates that only one evaluation report was 
completed during the 2001-02 fiscal year, it presents a balanced account of its 
key conclusions. The Solicitor General Canada report often refers to 
evaluation reports that are related to several strategic outcomes, but it does 
not provide information on their content. 


1.42 Identifying the level of resources allocated to achieve specific results is 
difficult. However, most of the reports managed to do so at the broad strategic 
outcome level. The Correctional Service Canada report provides a promising 
example of how to communicate this type of information. However, like other 
departments, it does not identify the resources allocated to achieve its key 
outputs and immediate outcomes. All the reports provide resources that are 
allocated according to business lines. Only one report identifies the financial 
contribution of each business line to each strategic outcome. It does this by 
equating its business lines to its strategic outcomes. For the most part, the 
reports do not provide financial information at the next level down for 
strategic outcomes or business lines namely, for major programs. 


Challenges to reporting outcomes 


1.43 We observed that some information about what activities were 
undertaken to produce the outputs or contribute to the outcomes is 
presented as the actual measures of outputs or outcomes. For example, the 
Citizenship and Immigration Canada report presents descriptions of recent 
legislative changes as measures of its performance. Other reports also present 
activities such as meetings, organized events and completed studies as results. 
While useful and relevant, any information on activities should also be 
accompanied by explanations about how they contribute to the results that 
are achieved for Canadians. 


1.44 As we indicated earlier, providing highlights of key findings from 
relevant evaluations and audits helps to explain the department's 
performance and the challenges it faces. We noted that Transport Canada’s 
report does not provide this type of information, although such evaluations 
and audits were in fact conducted. A number of these evaluations and audits 
address issues that are of significant interest to Canadians. It would have been 
helpful if the Department had informed Parliament about its efforts to address 


these issues. 


145 As previously noted, the Immigration and Refugee Board report 
provides a lot of information about its outputs. However, data on the numbers 
and proportions of claims accepted, rejected, withdrawn or abandoned 
through the refugee protection process are not provided, although they would 
be of interest to parliamentarians. Information on the numbers and 
proportions of appeals allowed and dismissed through the immigration appeal 
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process would also be helpful. We also observed that the Board provides little 
information on its outcomes. 


1.46 The fourth criterion states that performance information should be 
credible and balanced. First, it focusses on the quality of performance 
information and the reliability of information sources as the basis for judging 
the credibility of the data. Second, it focusses on whether there is balanced 
reporting of good results and shortcomings, and whether the level of detail for 
key results is appropriate. This criterion is fundamental to good performance 
reporting. If all the elements of good reporting are in place, but the 
information provided is not credible or balanced, then the report will not be 
useful for parliamentarians or Canadians. 


Departments do not generally report balanced results 


1.47 The ratings for this criterion were between basic and fair. Overall, the 
reports did not achieve a good rating (Exhibit 1.6). 


Exhibit 1.6 Performance information is credible and balanced—Results for the nine reports rated 


Citizenship and Immigration Canada 
Correctional Service Canada | 
Department of Justice 

Immigration and Refugee Board 
National Defence 

National Parole Board 

RCMP 


Solicitor General Canada [LS 


Transport Canada 


Levels 1 2 3 4 5 
Basic Fair Good Very good _ Excellent 


Promising practices in reporting credible and balanced results 


1.48 The National Parole Board report explains the limitations of the data 
on post-warrant expiry recidivism. It mentions that the number of re-offences 
measured by this indicator is probably much larger in reality because of 
certain limitations of the data. The Immigration and Refugee Board reports 
the number of complaints filed against members of the Board. Also, sections 
of the RCMP report provide endnotes that contain information about the 
external data sources. 


Challenges to reporting credible and balanced results 


1.49. We noted that, overall, the reports tend to present only positive 
performance information rather than both successes and shortcomings. This 
tendency to focus only on “good news” produces reports that convey a one- 
sided view of departments’ performance. Consequently, a lack of balanced 
reporting means that potential users, such as parliamentarians, may be 
tempted to discount them as biased, incomplete or lacking credibility. 
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150 We also identified several examples of data that were of limited quality. 
The reports provide little information about how departments manage data 
quality, what the limitations of the data are, and what they intend to do to 
make improvements. 


1.51 Managing well to obtain improved results is a challenge. As noted in 
our December 2000 Report, Chapter 20, Managing Departments for Results 
and Managing Horizontal Issues for Results, there is a need for more 
widespread measurement of the results departments accomplish and the use 
of that information to improve programs. The previous four criteria are based 
on a more traditional approach to reporting performance, where the 
organizational context must be provided, the performance defined, measured, 
and reported against expectations, and the information must be of the highest 
quality. However, even if all of these conditions have been met, if feedback on 
past performance is not used to improve future results, then it may only be a 
reporting exercise for external accountability. A performance report is also an 
invaluable tool for departments to demonstrate that they have learned from 
their shortcomings and challenges, and that they are using this information to 
improve their future performance. This should also encourage balanced 
reporting. 


1.52 The fifth criterion expects that performance reports will show the ways 
in which performance information is used, including how it will be used to 
establish future performance expectations. It assumes that performance 
information is not simply collected to create reports. Rather, it will be used to 
help departments make strategic decisions, highlight lessons learned, and 
identify how weak performance will be corrected. The performance 
information should also provide a credible discussion about the capacity of a 
department to produce sustainable results by continuing to perform well in 
the future. 


The use of performance information is not demonstrated 


1.53 Most of the reparts we rated only achieved a basic level on the fifth 
criterion (Exhibit 1.7). When departments produce key results that are of a 
reasonable quality, they should be able to demonstrate how this information is 
used to help make decisions about how to improve future performance. 


Promising practices in reporting the use of performance information 


1.54. The National Parole Board report states that it has had difficulty with 
managing pardon workloads and that this has an impact on processing times. 
In response to the growing backlog of applications for pardons, the Board lists 
the activities it has undertaken to improve its performance in this area. 


1.55 The Department of Justice report provides a short overview of lessons 
learned and the challenges it is facing for each of its business lines. Although 
it might be a promising practice that other departments could adopt, the 
relevance of the lessons learned is not always clear. 
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Exhibit 1.7 Use of performance information is demonstrated—Results for the nine reports rated 


Citizenship and Immigration Canada 
Correctional Service Canada } 
Department of Justice 

Immigration and Refugee Board 
National Defence 

National Parole Board 

RCMP f{ 


Solicitor General Canada 


Transport Canada 


Levels 1 2 3 4 5 
Basic Fair Good Very good _ Excellent 


Challenges to reporting the use of performance information 


1.56 None of the reports we rated discuss the capability of departments and 
agencies to continue to meet performance targets in the future. The ratings 
suggest that this could be one of the most challenging aspects of performance 
reporting. Facing this challenge will require that good information be 
incorporated into the decision-making process and that the information be 
used to correct problems and to evaluate whether or not the department is 
heading in the right direction. 


The challenge of reporting horizontal issues 


1.57 Horizontal issues are programs and activities that span the mandates of 
several departments and agencies and that receive a contribution from each. 
Reporting on these issues to Parliament presents particular challenges for the 
government as a whole, as well as for each of the respective departments. 
Departments have to determine at which point their contribution to a shared 
outcome requires that it be reported to Parliament. The government has 
recently used Canada’s Performance 2002 to report on the results of some 
horizontal areas. 


1.58 We expected that performance reports would provide sufficient and 
credible information to Parliament about the department’s performance for 
significant horizontal issues. We also expected that departments participating 
in a horizontal activity would 


¢ describe their role in the issue and their contribution to the shared 


outcome, 
* identify the lead department, where appropriate, and 


* direct readers to the lead department’s performance report. 


Reporting on the safety and security of Canadians is incomplete 


1.59 As the government has stated on a number of occasions, the events of 
September 11, 2001 demonstrate the need for an extensive, government- 
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wide response to terrorism. Parliamentarians and Canadians need to know 
how departments and agencies are working together in a horizontal manner 
to help protect Canadians from organized criminal and terrorist threats. They 
also need to know the value they are getting for government expenditures, 
such as the additional $7.7 billion allocated in the December 2001 federal 
Budget to this issue over the next five years. 


1.60 In order to measure the results of departmental contributions to this 
particular horizontal area, it would be useful to clearly define what these 
activities are supposed to achieve. Current descriptions of activities in this 
area at the government-wide level are sketchy. The Government of Canada’s 
Web site states that many government departments and agencies work 
together to keep Canada safe but it does not identify these organizations. 
Information is also lacking about departmental responsibilities related to 
organized crime—a key issue discussed in the 2001 Speech from the Throne. 


1.61 Government-wide reporting on the results of these activities is being 
developed. In Canada’s Performance 2002, the President of the Treasury Board 
identified “security from organized criminal and terrorist threats” as one 
horizontal area of government activity. The report refers the reader to the 
performance reports of the relevant departments. The Treasury Board 
Secretariat Web site identifies the departments that contribute to helping 
protect Canadians from organized criminal and terrorist threats, and provides 
Web links to their performance reports. 


1.62 Since government-wide documents provide only general references to 
horizontal issues, we expected to find more information on the participants 
and the results of their activities in this horizontal area in the departmental 
performance reports. Our rating model identifies two opportunities for 
departments to address horizontal issues in their performance reports: identify 
their role in the initiative and identify their contribution to the shared 


outcome. 


1.63 Eight of the nine reports reviewed list some departments’ key partners 
for their activities that address organized criminal and terrorist threats. 
However, the details vary significantly. Transport Canada simply lists all of its 
partners, without identifying the horizontal issue. Only three reports identify 
which department leads a particular horizontal issue. The Department of 
Justice provides information on horizontal, government-wide issues in a table 
that names the issue, the lead department, its partners, and the department's 


activities. 


1.64 Although we did not rate the Privy Council Office’s performance 
report, we noted that it addresses issues related to the security of Canadians. 
It states that, as part of the aftermath of September 11, 2001, it “led and co- 
ordinated the interdepartmental response to the crisis through the creation of 
formal co-ordination mechanisms and enhanced direct contact between 
personnel in different federal departments and agencies.” This report does 
not mention which departments were involved in the horizontal area or what 
their responsibilities were. 
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1.65 None of the performance reports indicate the contributions their 
partners have on the shared outcome. Results of activities designed to help 
protect Canadians from organized criminal and terrorist threats are not 
reported. Only the Solicitor General Canada refers to an evaluation of the 
Integrated Proceeds of Crime Initiative, but the report does not provide the 
actual results of its contribution to the shared outcome of that initiative. A 
few departments report on some of the challenges that they face in working 
with other departments to produce shared results. 


1.66 We noted that the Treasury Board Secretariat, in its Departmental 
Performance Reports 2002 Preparation Guide, requires departments to reflect 
the contribution their partners make to help them achieve their strategic 
outcomes. However, this requirement does not really go beyond asking them 
to identify their partners. This means that overall, there is little collective or 
systematic reporting to Parliament on the various initiatives aimed at 
increasing Canadians’ security and safety or on who is accountable for these 
efforts and their results. 


1.67 In our December 2000 Report, Chapter 20, Managing Departments for 
Results and Managing Horizontal Issues for Results, we recommended that 
the Treasury Board Secretariat should play a stronger leadership role in 
ensuring that horizontal issues are reported. We continue to support this view. 


168 Recommendation. The Treasury Board Secretariat should strengthen 
its guidance and actively encourage departments to ensure that their 
performance reports 


* identify significant horizontal areas, 
* identify the lead department, where appropriate, 
* identify the expected outcome for each area, and 


* indicate how the department’s activities and outputs contribute to the 
horizontal area’s shared outcomes. 


Treasury Board Secretariat’s response: The Treasury Board Secretariat will 
continue to encourage departments to report on horizontal initiatives. We 
will be strengthening our guidance on the preparation of departmental 
performance reports with respect to programs and initiatives that cut across 
departmental boundaries. 


Conclusion 


1.69 This is the first time that we have used our model more extensively to 
rate performance reports. While we have identified some promising practices 
and challenges faced by the nine departments, overall, we found that their 
progress in improving performance reporting to Parliament remains too slow. 
Our ratings also showed that the importance of reporting on horizontal issues 
is not well recognized. 
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1.70 These findings are not new. In fact, they continue a trend of audit 
observations. In 1988 we concluded that reports to Parliament did not 
provide a fully satisfactory basis for accountability. In 1992 we again indicated 
that these reports did not provide the necessary breadth of information. In 
1997 when the government reviewed its Expenditure Management System, 
we stated that progress in the area of performance reporting to Parliament 
was insufficient. In our 2000 Report, Chapter 19, Reporting Performance to 
Parliament: Progress Too Slow, we concluded that, “At the present pace it 
would take too many years for good performance to become routine.” 


1.71 In our 2000 report, we listed factors that contribute to the current state 
of reporting. These are as follows: 


* frequently, the basic principles of good reporting are not understood or 
applied; 


* performance reporting takes place in a political environment; and 


¢ there are no incentives available for good reporting practices, or 
sanctions applied for those that are bad. 


1.72 Various players can contribute to overcoming these obstacles. Of 
course, the departments should continue to work on improving their 
performance reports. The Treasury Board Secretariat should also continue to 
help them by challenging departmental performance reports and providing 
leadership and opportunities to increase their understanding and acceptance 
of the principles and practices of good performance reporting. 


1.73 Parliamentary reviews of performance reports would also encourage 
better reporting. Parliamentarians, through various standing committees, 
could demand clarification and explanations of performance expectations 
and information. They could also challenge departments and agencies. 


1.74 Our Office will continue to update and use the rating model as one 
tool to help encourage the improvement of performance reports to 
Parliament. By recognizing promising practices and identifying areas that 
should be improved, we hope to contribute to ensuring better reporting to 
Parliament. 


Government’s response: The government remains strongly committed to 
ongoing improvements to reporting to Parliament. In support of this, we 
welcome the efforts of the Auditor General to assess departmental 
performance reports in a systematic manner, identify strengths and 
weaknesses, and provide advice on how improvements can be made. 
Departmental performance reports reflect the responsibility of each minister 
and his or her department to account to Parliament for their performance 
against planned outcomes. We believe the findings, including the 
identification of promising practices, will help guide departments and 
agencies as they continue to improve their reporting to Parliament. 


As indicated in the 2003 Budget, in order to reinforce accountability and 
transparency in public reporting, the government will continue to improve 
the relevance, timeliness and clarity of the information it provides to 
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Parliament. This includes making greater use of electronic reporting on 
expenditures and the results achieved by government programs and activities. 
The government will continue to work with parliamentarians, parliamentary 
committees and the Auditor General to identify opportunities to improve 
reporting to Parliament. We will draw on the results of these discussions and 
the findings of this chapter, and on other sources, as we renew our guidance 
to departments on performance reporting. 


In particular, the recommendation that the Treasury Board Secretariat 
actively encourage departments to provide more fulsome reporting on 
significant horizontal initiatives, the Secretariat will continue to strengthen 
its guidance over the coming reporting period to underscore the importance 
of reporting on these horizontal initiatives. 
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About the Audit 


Objective 
The objective of this audit is to rate departmental performance reporting by 


¢ determining the overall quality of departmental performance reports, and providing Parliament and Canadians 
with information about departmental performance; and 


* determining if horizontal issues are properly identified in performance reports, using a shared outcome as an 
example. 


Scope 


We selected the performance reports of departments and agencies that have mandates that contribute to the horizontal 
area of helping protect Canadians from organized criminal and terrorist threats. This horizontal issue is one of those 
identified in the communities theme in the President of the Treasury Board’s Canada’s Performance 2002 report. 


We excluded the Canadian Security Intelligence Service and the Office of the Communications Security 
Establishment Commissioner because they are not required to prepare performance reports. The Customs program 
of the Canada Customs and Revenue Agency is also excluded, because although the Agency prepares a 
departmental performance report, it is one of the three agencies that report to Parliament through an annual report 
that this Office assesses separately. 


Grouping similar organizations together enables us to compare common issues and challenges. Although our model is 
designed primarily to assess performance reporting at the department level, it also takes horizontal reporting into account. 


Approach 
In this audit, we assessed nine performance reports against our rating model. 


These reports covered the period ending 31 March 2002. They were tabled in the House of Commons on 
7 November 2002. 


In the rating model, each of the five criteria is represented by a continuum that has five levels or stages of 
development. These are basic, fair, good, very good and excellent. An exceptional performance report would achieve 
an excellent rating in each criterion. 


The approach developed in our April 2002 Report, Chapter 6, A Model for Rating Departmental Performance 
Reports requires raters to validate each other’s ratings. With this in mind, our audit team used the model to rate the 
reports, the results of which were confirmed by experts and the Office audit team for each department and agency 
we examined. We did not audit each department’s systems and procedures for producing the information included in 
their reports. 


Audit team 
Assistant Auditor General: Maria Barrados 


Principal: Barry Leighton 
Director: Yves Genest 


Doreen Deveen 

_ Denis Jobin 
Catherine Livingstone 
Albert Melanson 

Paul Pilon 


For information, please contact Communications at (613) 995-3708 or 1-888-761-5953 (toll-free). 
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All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 


Table of Contents 


Main Points 


Introduction 


What we found in 2000 and 2001 
What has happened since 2001 
Focus of the follow-up 


Observations 


Compliance strategy 


Implementation of the Compliance Improvement Plan is progressing slowly 


Assessing risk 

Risk assessment still has gaps 

Targeting of high-risk air passengers has improved 

Pre-approval programs for low-risk travellers are moving ahead 
Targeting of high-risk commercial shipments needs further improvement 


Initiatives for low-risk commercial shipments not supported by a solid compliance verification regime 


Tools and training 


Improved system used to process travellers at airports 

Problems with system used to process travellers at land borders are unresolved 
Concerns remain about controlling commercial shipments in Windsor 

Interim measures being used to analyze results of compliance verifications 
New approach to verifying importers’ compliance is partially implemented 
Slow progress in improving customs officer training 

Compliance verification officer training has moved forward 


Improving border security 


Plans for spending public security and anti-terrorism funds 


Measuring performance 


Improvements to performance measures are at the planning stage 


Conclusion and Recommendations 


About the Follow-Up 


20 


22 


Report of the Auditor General of Canada—May 2003 Chapter 2 


Canada Customs and 
Revenue Agency 


Managing the Risks 
of Non-Compliance for Customs 


Main Points 


21 The events of September 11, 2001 have had a dramatic impact on the 
Canada Customs and Revenue Agency’s Customs program and the Agency 
has faced many challenges as it seeks to deter and detect non-compliance at 
the border. In our April 2000 and December 2001 reports, we made several 
recommendations to the Agency for managing the risks of non-compliance 
for its Customs program. We found that the Agency has made substantial 
progress in implementing two of our recommendations. That is, the Agency is 
now collecting advance passenger information from most airlines to help it 
target high-risk air travellers and it is using a new system at airports that 
allows better screening of travellers. 


22 In other cases, however, the Agency has only partially implemented our 
recommendations, even those that would help make the border more secure. 
We found that the Agency had developed a strategy for collecting 
information from other government entities about the risks that arise from 
the responsibilities it undertakes on their behalf and has started to collect this 
information. It has put in place an action plan to strengthen its compliance 
verification regime and has conducted a preliminary analysis of the results of 
completed compliance verifications to help identify high-risk areas of non- 
compliance. Until the strategy is fully implemented and a complete analysis is 
done, gaps will remain in the Agency’s risk assessments, and border 
management and post-release verification plans. 


23 The Agency has introduced tools to help customs officers determine 
their training needs. It has also developed new courses and continues to offer 
existing ones. However, customs officers continued to express concerns about 
their training and the Agency is unable to fully assess whether they have 
received the training they need. The Agency is developing an ambitious 
training program and long-term learning plan for its compliance verification 
officers in response to our December 2001 recommendations. 


24 The Agency has not made any significant progress in collecting the 
information that would enable it to tell Parliament whether its risk 
management strategy is working or to support its assessment of how well it is 
achieving its objectives for the Customs program. 


25 On 12 December 2001, the Canadian and American governments 
signed the Smart Border Declaration. The Declaration includes a 30-point 
Action Plan for Creating a Secure and Smart Border and the Agency is in charge 
of eight of the Action Plan’s initiatives. Two of these are NEXUS and Free 
and Secure Trade (FAST), which are binational programs that are intended 
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to make it easier for low-risk travellers and commercial shipments to cross the 
border. In 2001 we were concerned that the Agency was planning to go ahead 
with a similar Canadian program for commercial shipments before solidly 
establishing its compliance verification regime that ensures that the importers 
involved comply with trade laws and regulations. The Agency has worked 
hard to improve the regime, but, in our view, it is not yet solidly established 
and the Agency needs to proceed with caution with its programs for 
commercial shipments. 


26 We also found that the Agency has made progress implementing the 
other initiatives in the Action Plan. It has set up joint targeting units with 
United States customs officials to target in-transit marine containers, signed 
an agreement with the United States to share information on customs fraud, 
and completed a joint review of customs and immigration practices at 
international seaports. It has discovered some legal issues related to 
establishing joint border facilities with the United States and is studying 
them. As well, it is putting in place the mechanisms to enable the sharing of 
advance passenger information. 


27. The Agency was allocated $433 million over six years as part of the 
government's public security and anti-terrorism initiative. It analyzed why it 
needed the funds and what it expects to achieve with them, but it has not 
systematically tracked the actual spending. 


The Agency has responded. The Agency agrees with our recommendations 
and has indicated the actions it has planned or has under way to address 
them. 


2 Chapter 2 


Report of the Auditor General of Canada—May 2003 


CANADA CUSTOMS AND REVENUE AGENCY—MANAGING THE RISKS OF NON-COMPLIANCE FOR CUSTOMS 


Introduction 


28 In April 2000 we reported the results of our audit of how the Canada 
Customs and Revenue Agency manages the risks at ports of entry for 
travellers entering Canada. In December 2001 we reported the results of our 
audit of how it manages the risks of non-compliance for commercial 
shipments entering the country. We identified several risk-management 
problems in both audits. This follow-up provides a status report on how well 
the Agency has improved its efforts to manage these risks. The Office also 
recently completed an audit that included how Citizenship and Immigration 
Canada manages immigration risks at ports of entry. The results of that audit 
were reported in the Auditor General’s 2003 Report, Chapter 5, Citizenship 
and Immigration Canada—Control and Enforcement. 


29 The Customs program is an important part of the Canada Customs and 
Revenue Agency. The Agency’s 2002—03 Report on Plans and Priorities says 
the expected outcome of the Customs program is that “Canadians’ health, 
safety, security, and business interests are protected, and Canada’s economic 
growth is supported, through responsible border and trade management.” 


210 The Agency spends about 16 percent of its resources on the Customs 
program (Exhibit 2.1). Spending for the Customs program increased in 
2002-03. However, it is projected to fall in the next two years, while the 
Agency’s total planned spending will increase. About 8,300 people are 
employed by the Customs program, more than half of whom are uniformed 
customs officers. However, this number is expected to fall by 2004-05. 


What we found in 2000 and 2001 


211. Our 2000 audit found that the Agency was modernizing its Customs 
operations and using new technology. However, we found that it did not have 
an overall compliance strategy for its Customs program. We also found that 
while the Agency was using a risk management approach to determine which 
travellers present the greatest danger to Canadian society, its risk assessment 
was incomplete because it lacked important information from other 


Exhibit 2.1 The Agency’s resources for its Customs program and other business lines* 


Planned me Planned 


ee PGie ee a) ichel Actual Planned 
2000-01 2001-02 2002-03 2003-04 2004-05 
enane Ces a millions): 495.9 ; 560.2 BOD es HSSe2 527.6 
All business lines ($ millions) 6, HOMES Sees 3,363.8 3,364.5 3,431.9 
Full-time equivalents—Customs services 7,206 7,348 8,398 8,174 8,075 
Full-time equivalents—all business lines vey Naf: 45,729 AN Ja) \l 47,498 48,028 


* The Agency’s other business lines are tax services, benefit programs and other services, appeals, and corporate management and direction. 
** The planned spending for 2002-03 includes funding found in Budget Plan 2001 for public security and anti-terrorism initiatives. 


Source: Canada Customs and Revenue Agency, Annual Reports to Parliament and 2002-03 Report on Plans and Priorities 
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departments and agencies. Finally, we found that customs officers at land 
borders and airports needed to be better equipped and trained to do their 
complex jobs. 


212 Our 2001 audit found that the Agency had made it easier for importers 
to bring legitimate goods into Canada and had provided them with several 
options to process commercial shipments. However, it needed to improve its 
enforcement activities, specifically, identifying high-risk commercial 
shipments and analyzing the outcome of its activities. We also found that, 
after six years and a lot of effort, the Agency had made little headway in 
assessing how well importers were complying with Customs trade laws and 
regulations. However, the Agency was working on a revised approach that 
looked promising. 


2.13 The House of Commons Standing Committee on Public Accounts held 
a hearing on our 2001 audit and issued a report on 12 December 2002. The 

Committee echoed many of our concerns and made six recommendations to 
the Agency. The government will be responding to the Committee’s report in 


May 2003. 


What has happened since 2001 


214 The events of September 11, 2001 had a dramatic impact on the 
Customs program. Within hours of the attacks, ports of entry across Canada 
were placed on high alert, and customs officers were mobilized to process the 
many U.S.-destined planes that were diverted to Canada. In the days and 
weeks following the attacks, the Agency increased the screening of travellers 
and shipments to detect potential terrorist activities. 


2.15 Since then, Canadian and American government departments and 
agencies have strengthened their partnerships to improve security and other 
services at the border, while facilitating the flow of legitimate people and 
goods. On 12 December 2001 the Smart Border Declaration was signed by 
both governments. The Declaration includes a 30-point Action Plan for 
Creating a Secure and Smart Border that is supported by these four pillars: 


* the secure flow of people, 
* the secure flow of goods, 
* asecure infrastructure, and 


* co-ordination and information sharing in the enforcement of the plan’s 
objectives. 


2.16 The Agency is in charge of implementing the following initiatives of 
the Action Plan: 


* putting in place a single, alternative inspection system called NEXUS to 
enable pre-approved travellers to easily cross the border; 


* sharing advance passenger information and passenger name records for 
air travellers; 


* ensuring customs and immigration practices are in place at international 
seaports, including ferry terminals; 
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* putting in place harmonized commercial processing for the two-way 
movement of commercial goods from pre-approved traders; 


* processing truck and rail cargo at locations away from the border to 
improve security and facilitate trade; 


* creating joint border facilities in remote locations; 


¢ sharing information on customs fraud with the United States and 
sharing other customs data under the terms of the North American Free 
Trade Agreement; and 


* working with the United States Bureau of Customs and Border 
Protection to target high-risk marine containers that arrive at Canadian 
seaports on their way to the United States or that arrive at American 
seaports on their way to Canada. 


217. As part of its public security and anti-terrorism initiative announced in 
the 2001 Budget, the government set aside $433 million in funding over six 
years for the Agency to develop projects to increase the security of 
Canadians. Some of these projects have been expanded to binational 
initiatives in the 30-point Action Plan. Our Office intends to report in 2004 
on the additional public security and anti-terrorism funding and how it was 
spent. 


2.18 Since September 11, 2001, the Agency has faced many challenges and 
has spent a lot of time reviewing its border operations. Some of the initiatives 
that were part of the Investing in the Future: The Customs Action Plan 2000-— 
2004 are being implemented sooner than originally planned. In implementing 
the Smart Border Declaration, the Agency is focussing on increasing the level 
of co-operation with the United States and on binational programs. Border 
management is evolving quickly as both countries seek to improve security 
without impeding the free flow of legitimate goods and people. 


Focus of the follow-up 


219 The objective of our follow-up audit was to determine the progress the 
Agency has made in addressing the recommendations we made in our 

April 2000 and December 2001 reports and implementing the eight 
initiatives in the Smart Border Declaration’s 30-point Action Plan for which 
Customs has the lead role. Although our previous audit work was completed 
before September 11, 2001, many of our recommendations were aimed at 
improving the way the Agency manages the risks it faces at the border and 
measuring how effectively this is done. We also wanted to determine whether 
the Agency had appropriately planned how it will spend its $433 million 
portion of the $7.7 billion set aside in the 2001 Budget under the 
government’s public security and anti-terrorism initiative. The scope and 
criteria for this audit are set out in About the Follow-Up at the end of this 


chapter. 
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Compliance strategy 


Assessing risk 
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Observations 


Implementation of the Compliance Improvement Plan is progressing slowly 


2.20. In 2000 we found that the Agency did not have a compliance strategy 
for its Customs program that described its plans to encourage voluntary 
compliance, or that outlined its approach to responsible enforcement. A well- 
articulated, comprehensive strategy is a key element for promoting 
compliance and managing risk intelligently. 


2.21 In 2001 we found that the Agency had introduced its Compliance 
Improvement Plan. The plan laid out a systematic method of managing the 
Customs program that is based on risk. It was supported by a border 
management plan, a post-release verification plan, and a client services plan. 
The Agency knew that it would be several years before the Compliance 
Improvement Plan reached its full potential. In particular, there were many 
information gaps that needed to be filled. 


2.22 The 2002-03 Compliance Improvement Plan outlines the following: 


* national customs priorities for border examinations of travellers and 
commercial shipments; 


* compliance verifications of how well importers are following trade laws; 
* client service activities to promote compliance, and 


* investigations of smuggling and fraud. 


Border examinations are conducted at ports of entry and focus on threats to 
Canadians’ health, safety, and security, such as terrorism and contraband. The 
other activities outlined in the plan are usually done away from the border. 


2.23 One of the goals of the Compliance Improvement Plan is to collect and 
analyze information about compliance trends so that the Agency can identify 
and address non-compliance appropriately. However, as the Agency stated in 
its annual report for the year ending 31 March 2002, it has not made as much 
progress with the Plan as it had expected. For example, it is still studying 
which baseline measures would be best for measuring and reporting on overall 
compliance levels. As well, information gaps continue to make it difficult to 
do the analysis. We encourage the Agency to press ahead with this initiative. 


2.24 When developing a compliance strategy it is important to understand 
the risks, determine how significant they are, and find the best way to control 
them. Every traveller and shipment that enters Canada presents a risk, 
although in most cases the Agency considers that risk to be low (Exhibit 2.2). 
In order to reduce delays as much as possible, the Agency’s main focus at the 
border is checking health, safety, and security risks. Even so, examining each 
traveller and shipment is neither desirable nor practical. The Agency uses 
various techniques to focus on the travellers and shipments that present the 
greatest risk. In our April 2000 and December 2001 reports, we described 
some of these techniques, identified certain weaknesses, and recommended 
ways to improve them. 


Report of the Auditor General of Canada—May 2003 


CANADA CUSTOMS AND REVENUE AGENCY—MANAGING THE RISKS OF NON-COMPLIANCE FoR CUSTOMS 


ee a er ee ee A es oe 
Exhibit 2.2 Every traveller and shipment that crosses the border presents a potential risk 


Inadmissible people who will enter Canada: 


¢ illegal migrants; 
* terrorists; 
* people with criminal records. 


Illegal goods (contraband) that will enter Canada: 


* cocaine, heroin, ecstasy, cannabis, and others; 
* pornography, especially child pornography; 
* weapons and firearms. 


Restricted goods or goods needing permits that will enter Canada: 


¢ alcohol and tobacco; 

* endangered species; 

* textiles, milk protein, and prescription drugs; 

¢ hazardous waste; 

* contaminated goods, like those with foot and mouth disease. 
Travellers, importers, carriers, and drivers that make incorrect declarations (innocently 
or intentionally): 


* incorrect answers to questions posed by customs officers; 


* incorrect description of goods, including country of origin, quantities, value, end- 
use, and tariff classification (important for targeting). 


Risk assessment still has gaps 


2.25 Customs performs a front-line function at ports of entry, enforcing its 
own laws and parts of those of other government entities. These include 
Citizenship and Immigration Canada, the Canadian Food Inspection Agency, 
and Environment Canada. In 2000 we found that the Agency’s assessment of 
the risks that exist at ports of entry did not include information from other 
government entities about the risks that arise from the responsibilities it 
undertakes on their behalf. This year we found that the Agency had 
developed a strategy for an organized approach to gathering this information 
and using it in its Compliance Improvement Plan. We also found that it has 
started to collect information on other government entity risks and 
incorporate it into its border management and post-release verification plans. 
The Agency needs to implement the strategy in a timely manner so that its 
plans include the risk priorities of other government entities to help its 
officers focus their efforts on the areas of highest risk. In the longer term the 
Agency needs to integrate information on other government entity risks into 


its overall risk assessments. 


226 In 2000 we recommended that the Agency complete or update, on a 
priority basis, memoranda of understanding (MOU) with other government 
entities on whose behalf it acts. This is important to help ensure roles and 
responsibilities are understood and information is communicated in a timely 
fashion. Discussions have since taken place and the Agency has told us that it 
plans to complete or update most key Customs-related MOUs with other 
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government entities by 31 March 2005. We note that MOUs have already 
been signed with five other entities. This includes the MOU with Citizenship 
and Immigration Canada that was signed on 6 March 2003. 


Targeting of high-risk air passengers has improved 


2.27 Customs uses targeting procedures to identify high-risk air travellers 
before they arrive in Canada. These procedures include comparing passenger 
names with databases that contain intelligence information, as well as 
information on previous Customs or immigration infractions. 


2.28. In 2000 we noted that some Customs targeting units received advance 
passenger information for air travellers on an ad hoc basis, at the carrier’s 
discretion. We recommended that the Agency make obtaining advance 
passenger information a high priority. 


2.29 Legislation was passed in 2001 that requires airlines to provide advance 
passenger information and passenger name records to the Canada Customs 
and Revenue Agency. Advance passenger information includes the full name, 
date of birth, citizenship, and a few travel document details. The passenger 
name record includes more detailed travel information such as all the 
destinations where the traveller flew and how the ticket was paid for. In 
October 2002, the Agency started collecting advance passenger information 
and it expects to begin collecting passenger name records in the summer 

of 2003. It is using this information to identify high-risk passengers and plans 
to keep it in a database for six years to help it and other agencies identify 
security, public health, and criminal threats. The Privacy Commissioner of 
Canada had raised concerns about the Agency’s plans to keep the 
information, but these have been resolved. 


2.30 The Agency also set up passenger targeting units at eight major 
Canadian airports to review information received and target high-risk 
passengers. Targeted passengers usually go through a secondary examination 
that entails more extensive questioning. Citizenship and Immigration Canada 
officers also work in three of these units and are expected to be working in all 
eight by the spring of 2003. The Agency and Citizenship and Immigration 
Canada are also working with the United States Bureau of Customs and 
Border Protection to share information on high-risk travellers. For example, 
joint passenger analysis units were set up at the Vancouver International 
Airport and the Miami International Airport in September 2002. These are 
pilot projects and will be evaluated in the spring of 2003. 


2.31 When we visited Canadian airports in December 2002, we noted that 
customs officers were receiving limited advance passenger information from 
the airlines. Several airlines were still testing their ability to electronically 
transmit the information to the Agency. In the meantime, some airlines were 
faxing the information to the Agency. By March 2003 the situation had 
improved so that most airlines were transmitting their information 
electronically. However, one major airline was transmitting information about 
North American flights but not about most international flights. Until the 
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Agency receives passenger information electronically on the vast majority of 
passengers arriving in Canada, the targeting program will be constrained. 


Pre-approval programs for low-risk travellers are moving ahead 


232 A key objective of the Customs Action Plan 2000-2004 and the Smart 
Border Declaration is to pre-approve low-risk travellers to enable them to 
cross the border quickly so that customs officers can focus on other travellers 
who may be high-risk. There are two pre-approval programs for travellers: 


* CANPASS—a Canadian program, and 
* NEXUS—a joint Canadian-American program. 


2.33 Land borders. In 2000 we reviewed the Agency’s experience with the 
CANPASS-Highway program and noted that its potential had not been 
achieved. CANPASS-—Highway has now been replaced by NEXUS Highway. 
Travellers who receive a NEXUS Highway card are considered low-risk by 
Canada and the United States and can use their card to enter either country. 
NEXUS Highway was first introduced at the Blue Water Bridge (Sarnia—Port 
Huron) in 2000 and was expanded to other locations in British Columbia and 
Ontario in 2002-03. The Blue Water Bridge pilot project was evaluated and 
the results were encouraging. The Agency needs to firm up its evaluation 
plans for the new locations so that it can be sure that the program’s objectives 
are being met. 


2.34 Airports. While CANPASS—Highway has been replaced with NEXUS 
Highway, the Agency plans to go ahead with its CANPASS—Air program for 
air travellers. This program had been piloted for several years at the 
Vancouver International Airport but had little success. A CANPASS—Air 
enrolment centre was opened in Vancouver in March 2003. The plans call for 
centres to be opened in other airports sometime in the future. The Agency is 
also planning to introduce NEXUS Air as a one-year pilot project at the 
Ottawa and Montréal airports in 2004. 


2.35 The Agency has not conducted feasibility studies for either the 
CANPASS-Air initiative or the NEXUS Air initiative, although it is 
planning to conduct a marketing survey. We are concerned about this because 
such studies are needed to show whether these initiatives would be a 
worthwhile investment. We noted this same concern in 2000 with respect to 


the CANPASS-Highway program. 


Targeting of high-risk commercial shipments needs further improvement 


2.36 Identifying which commercial shipments present the highest risk of not 
complying with Canadian laws is an important part of enforcing those laws. It 
also allows low-risk shipments to enter Canada more easily. In 2001 we found 
some weaknesses in the targeting process, and recommended improvements. 
The Agency agreed to work on the improvements, provided the necessary 
systems and financial resources were available. 


2.37 This year we found that few improvements had been made to address 
our concerns about targeting high-risk commercial shipments. At the same 
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time, the Agency has developed and delivered a course on targeting 
commercial shipments and has improved its targeting of in-transit marine 
containers. At one land border port, we also noted that there was an 
increased emphasis on targeting high-risk shipments and providing the 
necessary resources to do so. However, at another land border port, we noted 
that the targeting function had been eliminated and that customs officers 
were relying on information provided by the Agency’s Contraband and 
Intelligence Services Directorate to target high-risk shipments. 


2.38 We also found that the number of commercial shipment examinations 
had doubled between 1 April 2001 and 31 March 2002 over the previous 
year. This included an increase of 50 percent in the number of random 
examinations. However, we found that Customs did little to analyze the 
results of these random examinations, which could help it improve its 
targeting. As well, customs officers again told us that, if other operational 
priorities intervene, they pay less attention to conducting random 
examinations. This too could diminish the usefulness of random 
examinations in identifying potential high-risk targets. 


2.39 Marine benchmarking study. One of the initiatives in the Smart 
Border Declaration is to review customs and immigration practices at 
international seaports, including ferry terminals. Officials from the Agency, 
Citizenship and Immigration Canada, the United States Customs Service, 
and the United States Immigration and Naturalization Service conducted the 
study and reported the results in May 2002. The objective of the study was to 
identify best practices, strengths, and weaknesses, and make 
recommendations. The next step in this project is to develop an action plan 
to address the recommendations. 


2.40 ‘Targeting marine containers. Many of the containers arriving at 
Canadian seaports are going to American destinations. Similarly, some 
containers arriving at American seaports are going to Canadian destinations. 
As part of their joint 30-point Action Plan, Canada and the United States are 
working to improve container inspection by targeting in-transit marine 
containers. Joint targeting units were set up in 2002 in Halifax; Montréal; 
Vancouver; Newark, New Jersey; and Seattle, Washington. At Canadian 
ports, American officers target containers going to the United States and 
these are examined by Canadian customs officers. At American ports, 
Canadian officers target containers going to Canada and these are examined 
by United States customs officers. Unit officers told us that the ability to 
exchange information helped them target high-risk containers. This 
initiative’s effectiveness at finding inadmissible people or contraband has not 
been evaluated. 


241 Sharing information on customs fraud. The 30-point Action Plan 
includes an initiative for both countries to share information about customs 
fraud. An agreement has been signed that sets out the terms and conditions 
under which such information can be shared. For example, the agreement 
states that only information on specific importers suspected of having 
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committed customs fraud in either country can be shared. To date, neither 
country has requested information covered in the agreement. 


Initiatives for low-risk commercial shipments not supported by a solid compliance 
verification regime 


242 A key objective of the Customs Action Plan 2000-2004 and the Smart 
Border Declaration is to enable low-risk commercial shipments to cross the 
border quickly so that customs officers can focus on other shipments that may 
be high-risk. This is accomplished by pre-approving importers, transportation 
companies, and drivers. There are two pre-approval programs for commercial 
shipments: 


* Customs Self Assessment (CSA)—a Canadian program, and 


¢ Free and Secure Trade (FAST) 


a joint Canadian-American program. 


243 Customs Self Assessment program. At the time of our 2001 audit, 
the Agency was planning to implement the CSA program. The program 
allows pre-approved importers to bring their shipments across the border 
quickly and account for them more easily. Importers, transportation 
companies, and drivers must all be pre-approved to participate in the 
program. Once the participants are approved, there is limited checking of 
shipments at the border. Rather, the Agency relies on its compliance 
verifications to ensure that importers have properly accounted for the goods 
and that they have paid all taxes and duties owing. Compliance verifications 
are critical in a self-assessment environment. In 2001 we were concerned that 
the Agency was planning to implement the CSA program before it had solidly 
established its compliance verification regime. The Standing Committee on 
Public Accounts expressed similar concerns and recommended in 

December 2002 that the CSA program be delayed. 


2.44 However, the Agency had already gone ahead with the CSA program. 
To date, five large importers have been approved to use the program, as well 
as many transportation companies and drivers. These five companies import 
about 21 percent of the total value of all the goods imported into Canada. 
The Agency is targeting importer participation in the program, by 2005, that 
would represent 50 percent of the total value of all the goods imported into 
Canada. This would include about 1,000 large importers. As the participation 
rate in the CSA program increases, the risk of not discovering any non- 
compliance also increases if a solid compliance verification regime is not in 
place. 


2.45 Since our 2001 audit, the Agency has put in place the framework to 
strengthen its compliance verification regime and measures to analyze 
compliance results. It has started to implement a new, two-part approach for 
verifying importers’ compliance. However, in our view, the Agency has not 
yet solidly established its compliance verification regime (paragraphs 2.60 to 
2.67). We remain concerned that it cannot adequately ensure that pre- 
approved importers are complying with Canada’s trade laws and regulations. 


2.46 Free and Secure Trade program. The Canadian and American 
governments have introduced a binational program called FAST to make it 
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easier for low-risk commercial shipments to cross the border. Importers, 
transportation companies, and drivers must all be pre-approved by the CSA 
program to participate in FAST. Application centres have been set up at 
several border crossings. To date, two importers have been approved under 
the FAST program. As with the CSA program, we are concerned that the 
Agency is pressing ahead with FAST before its compliance verification 
regime has been solidly established. In our view, the Agency needs to proceed 
cautiously with the CSA and FAST programs. 


2.47 Moving the clearance process. The Smart Border Declaration’s 30- 
point Action Plan also includes an initiative to move the clearance process 
away from the border. For example, Canadian and American customs officers 
could conduct their operations in one country or the other, at a shared 
facility. The goal of this initiative is to reduce delays at the border. Discussions 
have taken place between the Agency and the United States Bureau of 
Customs and Border Protection. However, some legal issues need to be 
worked out before a plan can be presented to officials, approved, and put in 
place. 


2.48 Customs officers must administer Customs legislation and parts of laws 
that govern other departments and agencies. They must know about passport 
and visa requirements, as well as laws that cover a broad range of products 
and services. Customs officers who work at the primary inspection line must 
quickly decide whether to let travellers and commercial shipments go on their 
way, or send them for a secondary inspection where they will be questioned 
more extensively and, in some cases, examined. 


249 Given the complexity of customs officers’ jobs, it is vital that they have 
the information, tools, and training they need to make correct decisions 
quickly. In our 2000 and 2001 audits, we found that information was not 
consistently shared, systems that the officers relied on needed to be improved, 
and training was uneven. 


Improved system used to process travellers at airports 


250 ‘Travellers arriving in Canada by air are asked to complete a Customs 
Declaration Card before they arrive at Customs. An officer reviews their 
declaration and conducts a brief interview. 


2.51 In 2000 we found that one of the key systems that was used to help 
customs officers decide whether to send a traveller for a secondary 
examination was outdated and used less than 50 percent of the time. Since 
then, the Agency has installed the Integrated Primary Inspection Line 
System. This system is directly linked to databases at the Agency and at 
Citizenship and Immigration Canada so that customs officers have immediate 
access to “lookouts” and enforcement information. A lookout alerts customs 
officers to watch for particular individuals and to send them for a secondary 
examination. We found that officers were using this system for at least 

90 percent of the air passengers they processed. This is a significant 
improvement over 2000. 
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Problems with system used to process travellers at land borders are unresolved 


2.52 Travellers arriving in Canada at land border crossings remain in their 
vehicles while a customs officer interviews them. Declaration cards are not 
required at land border crossings. Travellers usually only need to show some 
form of identification. 


2.53 At many land border crossings, customs officers use the Primary 
Automated Lookout System—Highway to screen travellers. The system reads 
licence plates and compares them with a Customs database to check for 
previous enforcement actions and/or lookouts. Officers can also use the 
system to check for immigration lookouts. 


2.54 In 2000 we found that the system read licence plates correctly about 
70 percent of the time. For the remaining 30 percent, customs officers had to 
manually correct the recorded information before the system was able to 
compare it with the database. We recommended that the Agency explore the 
possibility of improving the system. Although the Agency discussed the 
reliability of the licence-plate readers with the manufacturer, we have found 
no improvement. This is a concern because correcting the information diverts 
the officer’s attention away from interviewing the occupants in a vehicle. 
This raises the risk that travellers that should be sent for a secondary 
examination are not. This risk is greater when the officer does not correct the 
licence plate reading. 


2.55 We also note that the Agency’s system is based on licence plate 
information and cannot automatically connect with Citizenship and 
Immigration Canada’s lookout system, which is based on personal 
identification information. Unless customs officers enter an individual’s name 
into the Primary Automated Lookout System, they are unlikely to know if 
there is an immigration lookout for that individual. But they are doing this 
less than 10 percent of the time. Instead, they rely on their judgment to 
decide whether to refer travellers to a secondary immigration examination. 


2.56 Finally, we note that the Integrated Primary Inspection Line System 
used in airports is also available at 11 highway crossings to process individuals 
travelling by bus. It is also being tested at two highway crossings to process 


commercial truck drivers. 


Concerns remain about controlling commercial shipments in Windsor 


2.57 In 2001 we were concerned that the facility for examining commercial 
shipments at the Ambassador Bridge in Windsor is about two kilometres from 
the bridge. Unless officers escorted a truck from the bridge to the facility, 
which rarely happened, the Agency could not be sure that the truck went 
directly to the facility, or that no one had tampered with the shipment. The 
Standing Committee on Public Accounts was also concerned about this 
situation. In December 2002 it recommended that the Agency develop a 
solution as soon as possible and that a report containing options, solutions, 
and an action plan be written and tabled in the House of Commons no later 


than 31 March 2003. 
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258 This year we found that the Agency had issued a bulletin on escorting 
high-risk shipments in November 2001. Officials told us that escorts are used 
in high-risk situations when there is a specific lookout for a truck or driver, or 
when a customs officer is convinced the shipment or the driver is high-risk 
and is unlikely to go to the examination facility. We believe this is the same 
process that was followed in 2001. 


259 The Agency plans to report on the Ambassador Bridge (as requested by 
the Standing Committee on Public Accounts) in the government’s response 
to the Committee’s report. It is expected to be tabled in the House of 


Commons in May 2003. 


Interim measures being used to analyze results of compliance verifications 


2.60 In 2001 we recommended that the Agency develop a national 
computer system to capture the results of all compliance verification 
activities. The Standing Committee on Public Accounts also stressed that the 
Agency should thoroughly assess its data and information systems 
requirements and take timely action to improve its information management 
systems for compliance verification. 


2.61 Since then, we noted that the Customs Branch has developed a 
detailed business case for a national compliance verification reporting system, 
which is expected to cost about $20 million over a five-year period. The next 
step in the process is to seek senior management’s approval for the system and 
its funding. 


2.62 As an interim measure, the Branch developed a spreadsheet-based 
process to capture the results of some of its verifications. In 2000-01 it 
conducted compliance verifications to check whether importers had classified 
and valued goods correctly. The Agency prepared a preliminary analysis of 
the results of 74 of these verifications that had looked at three high-risk 
commodities: textiles/apparel, steel, and footwear. The analysis showed an 
error rate that ranged from 25 to 31 percent for the classification of these 
commodities and from 19 to 27 percent for valuation. Agency officials 
indicated that based on these results, potential trade data errors could reach 
more than $11 billion for these commodities. In 2001 we also noted high 
error rates of 29 percent for classification and 15 percent for valuation for all 
commodities. Accurate trade data are important for negotiating and 
monitoring trade agreements and are used by businesses to make domestic 
and foreign investment decisions. 


2.63 We were encouraged that the Agency had done this analysis to identify 
compliance problems, and had used it to plan its compliance verification 
activities for 2002—03. But the Agency did other verifications in 2000-01. 
However, because the spreadsheet is not yet fully developed, the Agency did 
not analyze the results of those verifications to identify high-risk areas of non- 
compliance. As the new reporting system is developed and the results of more 
compliance verifications are collected and analyzed for subsequent years, the 
Agency should be in a better position to determine the degree of compliance 
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with trade laws and regulations, and whether compliance has improved or 
deteriorated. 


New approach to verifying importers’ compliance is partially implemented 


2.64 At the time of our 2001 audit, the Agency was piloting a new, two-part 
approach for verifying importers’ compliance because it had experienced 
great difficulty implementing the previous approach. This approach includes 
compliance assessment reviews and a new verification process. A compliance 
assessment review is designed to measure an importer’s level of compliance 
and to identify areas where the importer is most likely to be non-compliant. 
The new verification process involves detailed testing of identified high-risk 
areas to determine the extent of non-compliance. The testing of the 
methodology and most of the training of compliance verification officers in all 
regions were completed in November 2001. Interim guides were also provided 
to compliance verification officers to help them conduct the verifications. 


265 Since our 2001 audit, about 260 compliance assessment reviews have 
been completed. Officials have done a limited analysis of the results of those 
reviews. However, they were unable to tell us how many new verification 
processes had been conducted as a result of the risks identified by compliance 
assessment reviews. 


2.66 In February 2002, the Agency developed a comprehensive, three-year 
action plan for managing the implementation of compliance assessment 
reviews and the new verification process. The plan outlined several 
components that are still being developed, including the following: 


* creating the trade verification manual, 
* creating a verification report template, 


¢ developing time standards for compliance assessment reviews and the 
new verification process, 


* enhancing client profile procedures, and 


* developing a quality assurance framework. 


The action plan laid out the work needed to implement the new approach, 
which is critical to strengthening the compliance verification regime. 


267 Progress has been made in implementing compliance assessment 
reviews and the new verification process. However, it is too early to determine 
the effectiveness of the new approach since the results of verifications have 
not yet been collected, reviewed, or analyzed. As part of the action plan, the 
Agency indicated that it will evaluate the methodology for compliance 
assessment reviews and the new verification process in 2003—04. 


Slow progress in improving customs officer training 


2.68 Training provides customs officers with the knowledge and skills they 
need to do their complex jobs. The Agency has designed several courses for 
officers. They are offered at the Agency training facility in Rigaud, Quebec, as 
well as in regional offices and at ports. Some courses are mandatory, such as 
the eight-week induction course for new recruits. Officers are then expected 
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to take additional mandatory training, depending on the area in which they 
will be working. For example, there is a three-to-five day course for processing 
land or air travellers, and a three-week course for processing commercial 
shipments. Other mandatory and discretionary courses are also offered on 
several topics. 


2.69 In 2000 we were concerned that the training needs of all employees 
were not being assessed on a regular basis, that training plans were not being 
developed each year, and that the Agency did not keep complete training 
records. Since then, the Agency has introduced a competency profile for 
customs officers. This profile helps officers determine where they have 
training needs. We found that about 20 percent of officers had used the 
competency profile. 


2.70 We also found that about half of the officers have completed an 
individual learning plan. The plan, which is approved by the officer’s 
supervisor, identifies the formal and informal training that the officer would 
like to take to be able to do a better job. However, more than two thirds of the 
officers we surveyed indicated that they were not aware of the training 
courses being offered, and about half said that their supervisor did not take a 
proactive role in helping them determine their training needs. Better 
information about when and where courses are being offered and more active 
participation by supervisors would improve the learning plans. We also noted 
that the Agency does not have an accurate record of the courses its officers 
have taken. This means that it is unable to fully assess whether employees 
have received the training they need. 


271 In 2000 we recommended that appropriate training be provided in a 
timely manner. The Agency has continued to offer its existing courses and has 
prepared and delivered new courses to inform its officers about changes in 
laws, and to teach them how to use new equipment and systems, and how to 
deal with new threats. For example, over 60 percent of officers have received 
training on the recent changes to the Immigration and Refugee Protection Act. 
As well, the Agency recently developed a one-day training course on 
chemical, biological, radiological, nuclear, and explosive threats, which has 
been delivered to over 85 percent of its customs officers. At the same time, 
officers told us that they did not have sufficient training to detect fraudulent 
documents or use new drug identification tools and technologies. They also 
said that they had not received refresher training in key areas which they felt 
they needed to keep their skills up-to-date. Furthermore, many officers 
indicated that they did not receive the training they needed to do their jobs 
when they needed it. They also told us that training was often allocated on an 
ad hoc or preferential basis, or only if it did not require travel and overtime 
costs. In our view, training should be allocated based on need. 


2.72 The Agency is expanding its self-directed training courses to provide 
“just-in-time” learning as close to the workplace as possible. Training officials 
told us that officers who took these courses said that they had improved their 
understanding of the area they had studied. At the same time, about 

60 percent of officers told us that they were concerned about the 
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effectiveness of this type of training. Officers said that there are not enough 
computers or training hours available, there are constant interruptions, and 
there is little opportunity to ask questions and discuss the material with 
colleagues. We encourage the Agency to take these concerns into account as 
it develops more self-directed learning courses and particularly if the subject 
matter is complex. 


273 In 2000 we were concerned that students were not being trained 
consistently across the country. We noted that headquarters had developed a 
three-week course for new students, but that local ports sometimes shortened 
it to two weeks. We also noted that after classroom training, students were 
expected to “shadow” a customs officer. However, the time spent shadowing 
an officer varied from one day to one week. This year we found that students 
were still receiving two to three weeks of training and that the time spent 
shadowing varied from one location to another. In 2001-02 the Agency 
employed approximately 1,200 student customs officers. Because they make 
critical decisions at the primary inspection line, we remain concerned that 
the inconsistent training of students could pose an unnecessary risk for 
Customs. 


Compliance verification officer training has moved forward 


274 In 2001 we identified several concerns related to the training of 
compliance verification officers. Since then, we noted that the Agency has 
worked to improve training by doing the following: 


¢ In the spring of 2002, the Agency conducted focus groups in Calgary, 
Montréal, and Toronto with compliance verification officers who had 
various levels of expertise to determine their training needs. 


* In September 2002 the Agency drafted a long-term learning plan. The 
plan was approved by headquarters and regional compliance verification 
directors and is to be presented to the Customs management committee 
for approval by June 2003. 


2.15 The Agency outlined in its learning plan that it would develop and 
pilot an apprenticeship program; determine a core curriculum for beginner, 
intermediate, and senior compliance verification officers; and ensure that the 
training provided is appropriate and timely, and that it will be evaluated. 


2.16 To build the long-term training framework, the Agency has also 


* developed an introductory course for compliance verification officers 
and delivered several of these sessions, mostly to new officers; 


¢ developed, piloted, and finalized for national use a basic accounting 
course, as well as completed the related trainer’s course; 


* started to develop an intermediate-level accounting course for 2003-04; 


and 


* targeted the development of an advanced-level accounting course and 


an auditing course for 2004-05. 
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Improving border security 


2.71 Weare encouraged by these steps. However, we are concerned that the 
proposed development date for the introductory-level auditing course has 
been moved several times and is now targeted for 2004-05. Compliance 
verification officers need to have a sound knowledge of auditing techniques, 
in addition to accounting practices and trade regulations. We believe that the 
Agency needs to develop the auditing course sooner so that the training 
framework provides a solid foundation for training compliance verification 
officers. 


Plans for spending public security and anti-terrorism funds 


2.78 The federal government announced in its 2001 Budget that it would 
spend $7.7 billion between 2001-02 and 2006-07 to enhance the personal 
and economic security of Canadians. The goals of the spending were to keep 
Canadians safe, terrorists out of Canada, and Canada’s borders secure, open, 
and efficient. The Agency’s share of the funding was $433 million 


(Exhibit 2.3). 


Exhibit 2.3 How the Agency plans to spend its funding for public security and anti-terrorism ($ millions) 


2001-2002 2002-2003 | 2003-2004 2004-2005 | 2005-2006 | 2006-2007 Total 
Public Security—hiring and training 40 7 16 16 16 16 WAAL 
new customs officers 
Better tools for assessing risk and 
detecting inadmissible people 
e Advance passenger information 6 i 10 9 g 9 54 
¢ Integrated Primary Inspection 4 11 9 3 3 3 29 
Line System 
* Intelligence Management 2 2 2 1 1 1 9 
System 
Strengthening technology at 7 54 17, 16 14 14 WZ 
airports and seaports 
Customs-controlled areas at 5 5 5 4 4 4 Bil 
airports 
Better service to small business — 5 5 2 2 1 15 
a 
Sub-total 64 IOS 60 yl 49 48 OTe; 
Expediting pre-approved travellers* -- LS ES 12 8 8 58 
64 11/210) 715 63 Sy) ISX) 435 
Less: Allocation to Justice 2 
Agency total | i | a seo | ieee HN 
ee £4) S q | Pe 9 if x 


ae 


The December 2001 Budget included estimated costs for expediting pre-approved travellers that were originally intended to be full cost-recovery initiatives. 


The Agency plans to fund these initiatives through user fees and internal sources. 


Source; Canada Customs and Revenue Agency 
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279 The Agency has allocated this funding to the following projects: 


* hiring and training new customs officers to provide enhanced vigilance 
at all border locations; 


* developing and using the advance passenger information and passenger 
name record targeting and analysis tools; 


* acquiring additional equipment for the Integrated Primary Inspection 
Line System and installing the system at additional locations; 


* implementing the Customs Intelligence Management System used to 
identify high-risk individuals and organizations; 


* acquiring additional equipment such as x-ray machines to be used at 
international airports and seaports, as well as developing new systems to 
improve tisk assessment; 


* acquiring new equipment for areas in international airports that are 
controlled by Customs, and hiring and training new customs officers to 
work in those areas; 


* improving service to small businesses through new Internet applications 
and dedicated client services officers; and 


* expanding the NEXUS Highway program and developing the 
CANPASS-Air program to expedite pre-approved travellers. 


One of these projects—NEXUS Highway—is also an initiative in the 30- 


point Action Plan. The funding for another project—advance passenger 
information—is to support a Canadian implementation. This project has 
been expanded to a binational initiative in the 30-point Action Plan, and the 
Agency is seeking additional funding. Similarly, the funding for CANPASS-— 
Air is to support a Canadian implementation. This project has been expanded 
to NEXUS Air, a binational initiative in the 30-point Action Plan for which 


the Agency is seeking additional funding. 


2.80 The Treasury Board has approved the Agency’s original funding 
requests for all of the projects except the last one, expediting pre-approved 
travellers. The Agency plans to fund that project through user fees and 
internal sources. We reviewed the funding requests and found that the 
Agency had analyzed why the funds were needed and had set out the results 
the Agency expected to deliver. This was supported by a description of the 
risks, calculations of the resources needed to achieve the goals for each 
project, and any assumptions made in preparing the calculations. 


2.81 While it is too early to determine what has been accomplished to date, 
we found that the Agency has not systematically tracked its spending on 
these projects. We are concerned that it will not be able to fully account for 
how the funds have been spent. 


Improvements to performance measures are at the planning stage 


282 The expected outcomes of the Customs program are to protect 
Canadians’ health, safety, security, and business interests, and support 
Canada’s economic growth through responsible border and trade 
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management. We expected that the Agency would have performance 
measures in place to show how well it is achieving these outcomes. 


283 In 2000 and 2001 we noted that there were many gaps in the Agency’s 
performance measures for its Customs program. We recommended several 
improvements, including that the Agency should collect the information it 
needs to assess how effective its risk management approach is. 


2.84 Since we conducted our audits, the Canada Customs and Revenue 
Agency has presented two annual reports to Parliament. Under the Canada 
Customs and Revenue Agency Act, the Auditor General is required to assess 
the fairness and reliability of the performance information in the annual 
report. 


2.85 The Agency’s report for the year ending 31 March 2002 recognizes 
that its conclusions about the performance of its Customs program are based 
largely on a qualitative assessment and that more work is needed to develop 
comprehensive performance indicators. Nevertheless, the Agency concluded 
that it had mostly met its anticipated results for its Customs program. The 
Agency plans to address the weaknesses in its Customs performance 
measures. 


286 The Standing Committee on Public Accounts expects the Agency to 
develop a comprehensive performance measurement regime and to gather 
and use quantitative information to reinforce its management practices and 
accountability reporting. We agree with the Committee’s expectations. While 
we recognize that this is a long-term project, we expect to see progress each 
year. 


Conclusion and Recommendations 


2.87 Overall, the Agency is making satisfactory progress in implementing 
the recommendations from our 2000 and 2001 audits of the Customs 
program. It has substantially implemented two of our recommendations. That 
is, it is now collecting and using advance passenger information and it is using 
a new system to better screen air travellers. However, there is still much to be 
done in several critical areas. These include integrating the risks of other 
government entities into Customs risk assessments, analyzing examination 
results to provide better information for targeting high-risk travellers and 
shipments, analyzing compliance verification results to identify high-risk 
areas of non-compliance, and ensuring customs officers get the training they 
need in a timely manner. 


2.88 With respect to the Smart Border Declaration’s 30-point Action Plan, 
the Agency has made good progress in implementing the initiatives for 
targeting in-transit marine shipments, implementing the NEXUS program for 
low-risk travellers, and sharing information on customs fraud with the United 
States. As well, some progress has been made to implement the other 

five initiatives in the Action Plan. 
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289 The Agency has appropriately planned (except in the area of 
accountability) how it will spend its $433 million share of the funding for the 
government’s public security and anti-terrorism initiative. It has not 
systematically tracked its spending on these projects and will likely not be 
able to fully account for how the funds have been spent. 


2.90 Recommendation. The Agency should complete the implementation 
of the recommendations from our 2000 and 2001 reports in a timely manner. 


Agency’s response. We agree with the Office of the Auditor General that we 
are making satisfactory progress in implementing the recommendations from 


the Office’s 2000 and 2001 audits of our Customs program. 


The Agency agrees with the recommendation and will continue to work on 
implementing the action plans that have been prepared to respond to the 
two audits under review. 


The Agency has implemented NEXUS, advance passenger information/ 

passenger name record, Free and Secure Trade (FAST), and the Customs Self 
Assessment program as alternative ways to process travellers and commercial 
shipments. CANPASS—Air will be implemented in the summer of 2003. The 
Agency plans to monitor the results of these initiatives in the next fiscal year. 


2.91 Recommendation. The Agency should complete the implementation 
of the eight initiatives in the 30-point Action Plan for which Customs has the 
lead role, taking into account the observations made in this chapter. 


Agency’s response. The Agency agrees with the Office of the Auditor 
General that we are making good progress with respect to the Smart Border 
Declaration’s 30-point Action Plan. 


The Agency has made good progress in implementing the following 
initiatives: NEXUS Highway; NEXUS Air; advance passenger information/ 
passenger name record; FAST; and in the exchange of trade data with the 
United States Bureau of Customs and Border Protection; in the marine 
benchmarking study, which has been completed; and in targeting in-transit 
marine containers. We will continue to work on these initiatives to ensure 
that border security is maintained and enhanced. 


2.92 Recommendation. The Agency should track the spending and results 
achieved on the public security and anti-terrorism initiative. 


Agency’s response. We agree with the Office of the Auditor General that the 
Agency has appropriately planned how it will spend its $433 million share of 
the funding for the government’s public security and anti-terrorism initiative. 


The Agency agrees with this recommendation and will work toward keeping 
better track of Agency spending on the public security and anti-terrorism 
initiative. The Agency will improve the system in this regard. 
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About the Follow-Up 


Objectives 


To determine the progress the Canada Customs and Revenue Agency has made to address the recommendations we 
made in our April 2000 and December 2001 reports, and to implement the eight initiatives in the Smart Border 
Declaration’s 30-point Action Plan for which Customs has the lead role. 


To determine whether the Agency had appropriately planned how it will spend its $433 million portion of the 
$7.7 billion set aside in the 2001 Budget under the government’s public security and anti-terrorism initiative. 


Scope and approach 


Our follow-up audit covered the recommendations and supporting observations in our 2000 Report, Chapter 5, 
Travellers to Canada: Managing the Risks at Ports of Entry and our 2001 Report, Chapter 8, Managing the Risks of 
Non-Compliance for Commercial Shipments Entering Canada. We also reviewed the eight initiatives in the Smart 
Border Declaration’s 30-point Action Plan for which the Agency has the lead role and the Agency’s plans for 
spending the $433 million it was allocated under the government’s public security and anti-terrorism initiative. 


We interviewed Agency officials in Ottawa and in selected ports of entry. We also analyzed data provided by the 
Agency, and conducted a statistical survey of customs officers working at ports of entry to obtain their opinions on 
training. Our survey estimates at the national level are accurate within plus or minus 6 percent, 19 times out of 20. 


Criteria 


We expected that the Canada Customs and Revenue Agency would have made satisfactory progress in 
implementing our recommendations. 


The criteria from the 2000 and 2001 audits remain relevant. Therefore, we expected that Customs would have done 
the following: 
* put in place risk assessment and targeting processes that effectively differentiate between low-risk and high-risk 
travellers and commercial shipments; 


* appropriately planned and implemented its new initiatives in alternative ways to process travellers and 
commercial shipments, and monitored results to ensure that it is reaching its goals; 


* put in place mechanisms and processes to ensure that it is appropriately managing risks arising from the 
responsibilities it carries out on behalf of other government entities; 


* ensured that management and staff are able to acquire and maintain the knowledge and skills they need to 
fulfill their role in risk management; 


* clearly stated its risk management strategy and established appropriate criteria to assess the risks associated 
with the movement of people and goods across the border; 


* designed, implemented, and monitored a post-release verification regime to ensure compliance; 


* measured and reported on the results of its traveller and commercial programs for purposes of accountability 
and continuous improvement; and 


* designed an action plan for spending the public security and anti-terrorism funding that includes a needs 
analysis, expected results, resource requirements, risks, assumptions, and accountabilities for each project. 
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Assistant Auditor General: Douglas Timmins 
Principal: Jamie Hood 
Directors: Lilian Goh, John Pritchard, Katherine Rossetti 
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The Economic Component of the 
Canadian Immigration Program 


All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 
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The Economic Component of the 
Canadian Immigration Program 


Main Points 


3.1 In our 2000 audit on the economic component of the Canadian 
Immigration Program, we noted serious problems in the management and 
delivery of the Program. Many of the issues we raised in that audit have been 
addressed by the new Immigration and Refugee Protection Act and the 
Immigration and Refugee Protection Regulations. As most of the provisions 
came into effect 28 June 2002, it is still too early to assess the impact of the 
new act and regulations on some of the issues. 


3.2. Citizenship and Immigration Canada has taken a number of steps with 
regard to medical surveillance including establishing a Medical Surveillance 
Unit in a new Medical Services Branch. However, we are still concerned that 
the Department does not know what percentage of immigrants comply with 
medical surveillance requirements and within what time frame. It is 
important that the Department ensure that immigrants under medical 
surveillance report to the relevant public health authorities and thus comply 
with the conditions attached to their visas. We also identified a new issue— 
refugee claimants and public health authorities in the provinces and 
territories are not notified when a claimant requires medical surveillance for 


inactive tuberculosis. 


3.3 Although the Department is developing a method and strategy for 
quality assurance, we observed that a department-wide quality assurance 
framework to monitor the quality of selection decisions for the economic 
component has not been implemented. This framework is essential to ensure 
the consistency, fairness, and integrity of selection decisions. 


34 Many applicants apply to offices outside their country of residence, 
hoping that their wait will be shorter than in their own country. The new 
Immigration and Refugee Protection Regulations permit the Department to 
greatly reduce offshore processing. At the time of our field work, offices 
abroad were still accepting offshore applications. The Department has now 
issued instructions to offices abroad to implement the regulation dealing with 
offshore processing on | May 2003. 


3.5 The Department has made progress in how it controls revenue. The 
Department has developed a new Permanent Resident Card; however, 
problems still exist with controls over visa forms. It has made progress in 
updating information technology and has strengthened the internal audit 


function. 


3.6 The Department has made progress in improving its ability to 
determine inadmissibility for reasons of criminality and security. 
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37. The new act and regulations also address other issues we raised in 2000 


by: 


¢ barring applicants deemed inadmissible due to misrepresentation from 
re-applying for two years; 


¢ defining “excessive demand” for health and social services; and 


¢ allowing for application for non-disclosure of information during 
admissibility hearings, immigration appeals, and judicial reviews. 


3.8 The new act addresses some other areas; for example, criteria and tools 
for selecting skilled worker and business immigrants, and initiatives to assist 
visa officers in determining whether documents are fraudulent. However it is 
too soon for us to assess the impact of the legislation on these areas. 


3.9 In its reports to Parliament, the Department indicated that 137,119 
skilled workers entered Canada in 2001. We are concerned that this 
information may be misunderstood. The information would be more clearly 
presented as 137,119 skilled workers, 58,860 of that number being principal 
applicants and 78,259 their dependants. 


The Department has responded. Citizenship and Immigration Canada 
agrees with our recommendations and continues to act on recommendations 
from the 2000 audit that have not yet been fully implemented. 
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Introduction 


3.10 In our 2000 Report we examined the economic component of the 
Canadian Immigration Program. The economic component consists mainly of 
skilled workers and business immigrants. Business immigrants consist of 
investors, entrepreneurs, and the self-employed. 


3.11 The Program recruits highly qualified people who could adapt to our 
society and contribute to our economy. But the problems in the management 
and delivery of the Program that we noted in our Report limited Canada’s 
ability to make the most of the economic and social benefits that immigration 
affords. They also limit Canada’s ability to protect the integrity of the 
Immigration Program. This is our first follow-up to that Report. 


3.12 Since we tabled that Report the government passed the Immigration 
and Refugee Protection Act and the Immigration and Refugee Protection 
Regulations. This was the first major overhaul of the Immigration Act in some 
25 years. The Department’s senior managers were very involved in the past 
few years with the preparation and passage of the new legislation. Following 
extensive consultations with Canadians, the Department prepared a 
document in 1999 that served as the basis for discussion with the provinces, 
federal partners, and Canadians. The Act was passed in the fall of 2001, with 
most provisions coming into effect on 28 June 2002. 


3.13 On 31 December 2002 the Department had 91 service locations 
throughout the world to serve various geographic areas. For example, London 
is responsible for processing cases from the United Kingdom, Ireland, and the 
Nordic and Gulf countries. New Delhi is responsible for processing cases from 
India, Nepal, and Bhutan. 


Focus of the follow-up 


3.14 The objective of this follow-up was to determine what actions the 
Department has taken in response to the observations and recommendations 
made in our April 2000 Report on the economic component of the Canadian 
Immigration Program. We also audited medical surveillance procedures for 
refugee claimants in Canada. We reviewed two status reports prepared by the 
Department on actions it has taken or is planning. Further references in this 
chapter are to the second status report. Further details about the follow-up 
can be found in About the Follow-Up at the end of the chapter. 


Observations and Recommendations 


3.15 Many of the recommendations that we made in the 2000 Report have 
been dealt with by the new Immigration and Refugee Protection Act (IRPA) and 
its accompanying regulations (Exhibit 3.1). Each of these recommendations is 


discussed in more detail in the chapter. 


3.16 Immigration levels. Citizenship and Immigration Canada is meeting 
its planned overall immigration levels (Exhibit 3.2). The Department has 


Report of the Auditor General of Canada—May 2003 


Chapter 3 | 3 


CITIZENSHIP AND IMMIGRATION CANADA—THE ECONOMIC COMPONENT OF THE CANADIAN IMMIGRATION PROGRAM 


te § 
‘PIB 
VOGEL 


> danicinne 
On GECISIONS 


developed a resource allocation model to identify the level of resources it 


requires to process applications in offices abroad. Immigration uses the model 


to estimate the number of visa officers it needs to meet its planned 


immigration levels. 


Selection criteria have been amended 


3.17. In 2000 we recommended that selection criteria for choosing which 


economic immigrants Canada accepts be amended to make them more 


Exhibit 3.1 Recommendations from our April 2000 audit that were dealt with by the new /mmigration and Refugee Protection Act (IRPA) and its 


regulations 


Issue 


Selection criteria 


Recommendation 


The Department should amend 
selection criteria to make them 
conducive to rigorous selection and 
achievement of objectives. 


IRPA or regulations 


Regulations 73- 
109 


Description 


Criteria amended to allow for more 
objective assessment based on 
consistent standards, and the 
selection of immigrants based on 
ability to establish successfully in 
Canada 


Selection tools 


The Department should ensure that 
visa officers have the necessary tools 
and means to help them make their 
selection decisions efficiently and 
effectively. 


Regulations 78 
and 79 


Education points awarded based on 
having both a credential and a 
minimum number of years of 
education and formal training 


Skilled workers’ language proficiency 
assessed either by a designated third 
party organization or institution, or by 
other evidence submitted in writing 


Misrepresentation 


The Department should establish and 
implement a strategy to reduce to an 
acceptable and manageable level the 
risk that applicants will submit false 

statements or fraudulent documents. 


IRPA sections 
40(1) and 40(2) 


Applicants deemed inadmissible 
because of misrepresentation cannot 
re-apply for a period of two years 


Excessive demand 


There is a need to define the term 
“excessive demand” to ensure 
compliance with the Act. 


Regulation 1(1) 


Excessive demand defined as needing 
more than the average Canadian per 

capita cost for health care over 5-10 
years 


Protection of 
information 


The Department, with the RCMP and 
CSIS, should ensure that in determining 
admissibility, they have the tools to 
allow them to use information whose 
source or nature cannot be disclosed, 
while ensuring that applicants are 
treated fairly. 


IRPA sections 76, 
86, and 87 


Minister may make application for 
non-disclosure of information during 
admissibility hearings, immigration 
appeals, and judicial reviews 


Offshore processing 


The Department should ensure that 
applications are processed in the offices 
that have the necessary skills to make 
informed and consistent decisions. 


Regulation 11(1) 


Application for a Permanent Resident 
Visa must be made to the immigration 
office that serves the country where 
the applicant has been residing for at 
least one year or the applicant's 
country of nationality or habitual 
residence 
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objective and in line with the Immigration Program’s objectives. The 
Department stated that it was redesigning the selection system and the new 
model would shift from the current system that selects skilled workers based 
on their occupations to a system based on the workers’ sound and transferable 
skill sets. Selection criteria have been amended in the new act (Appendix). 
At the time of our field work, the Department’s experience using the new 
selection criteria was limited because the new act had only been in place 
several months. It is too soon for us to assess the impact of the new criteria on 
departmental operations. 


Tools developed and training for visa officers improved 


3.18 In the 2000 Report we noted that the lack of certain basic tools that 
would help visa officers in their decision-making hindered the efficiency and 
effectiveness of the Department’s operations. For example, officers used their 
individual judgments to evaluate applicants’ English or French. No tools 
existed to help officers readily assess education and employment training. 
Officers conducted their own research. They evaluated applicants’ language 
skills, academic training, and work experience based on interviews—one of 
the most costly and time-consuming activities in the processing of 
applications. 


3.19 In our 2000 audit we noted that the training of visa officers was 
insufficient in several respects. We recommended that the Department 
review its overall training strategy and ensure that all those responsible for 
immigration decisions receive appropriate training. 


Exhibit 3.2 Planned immigration levels and landings of principal applicants and dependants 


ee 7 i | 2001 2002 

a . | : : Planned levels S Cnvlites 3 Planned levels Landings Planned levels 
Eeonomie COnonenE . | | 
Skilled workers 100,500-113,300 118,463 100,500-113,300 NSH7/, AMS) 115,800-125,300 
Business 15,000-16,000 35655 15,000-16,000 14,579 12,000-13,000 
Others* 5,400 4,496 5,400 4,102 3,200-4,300 
Total economic 120,900-134,700 136,614 120,900-134,700 155,800 131,000-142,600 
component 
Family 57,000-61,000 60,515 57,000-61 ,000 66,647 56,000-62 ,000 
Refugees 22,100-29,300 26,747 22,100-29,300 26,530 23,000-30,400 
Kosovo refugees and - Sisc5 = ily Sie)S} - 
others 
Total AEE 200,000-225,000 227,209 200,000-225,000 250,346 210,000-235,000 


*Numbers from 2000 and 2001 have been restated to align with IRPA categories. Includes provincial and territorial nominees and live-in caregivers. 


Source: Citizenship and Immigration Canada 
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Arranged employment: An offer of 
indeterminate employment in Canada 


3.20 New selection tools developed. The Department has developed some 
new tools to assist officers assessing applications under the Immigration and 
Refugee Protection Act. In addition, the Regulations are designed to enable the 
visa officer to make an objective assessment. For example, the Act encourages 
economic applicants to submit a language test administered by a third party. If 
applicants do not submit this type of test result, then they must submit a 
written explanation and supporting documents to show evidence of their 
language ability. The Regulations state that Human Resources Development 
Canada can pre-approve arranged employment. The Regulations require 
officers to assess education based on standards that exist in the country of 
study. The standards consist of credentials and the number of years of 
education and formal training. At the time of our field work, the 
Department’s experience using the new selection tools was limited because 
the new act had only been in place several months. The Department expects 
that these tools will help reduce the need for interviews to assess language 
skills and education. 


3.21. Progress in training visa officers. The Department has spent a 
considerable amount of time and energy developing and improving training 
courses; for example, courses for local staff and courses on litigation 
management. Course evaluations that we reviewed indicated that courses 
were generally well received. The new act’s implementation required an 
extensive, co-ordinated training effort by the Department. Staff at each office 
abroad were selected to become trainers. They trained at departmental 
headquarters in Canada and on return to their offices abroad trained the 
remaining staff and provided ongoing support. Other initiatives included new 
manuals available electronically, messages on procedures distributed 
electronically, an IRPA intranet site established, and two IRPA help desks 
established for several months: one for calls from locations in Canada; the 
other for calls from offices abroad. Many officers rated highly the training 
they received on the new act although most indicated they would need to re- 
familiarize themselves with the course material before starting the new 
processing of skilled workers, which would be several months later. 


3.22 Satisfactory progress in litigation management. In some cases a 
judicial review by the Federal Court may direct Immigration to re-examine a 
refused application. Our previous audit noted an increase in such cases and 
the heavy demand on resources that resulted. The Department's status report 
noted several initiatives in this area. 


3.23. The Department has placed more emphasis on managing litigation 
issues. Initiatives include litigation management newsletters, annual visa 
litigation reports, a guide to cross-examination for visa officers, and form 
letters for accepting and refusing applications. For litigation management 
training, initiatives include a revised course for visa officers and joint training 
with the Department of Justice. Citizenship and Immigration Canada’s visa 
litigation reports for 2000 and 2001 indicate that judicial reviews have 
decreased slightly and the Department’s success rate in Federal Court has 
increased slightly since our last audit. 
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3.24 Tools and training for assessing business immigrants’ applications 
are being developed. Assessing business immigration applications requires a 
certain level of business knowledge and expertise. In 2001, 14,579 business 
immigrants, consisting of principal applicants and their dependants, landed in 
Canada. This is about nine per cent of the economic immigrants who landed 
that year. 


3.25 The Department is developing a tool that third parties will use to assess 
the business performance of applicants. Visa officers could then use this 
assessment as part of their overall evaluation of the application. At the time 
of our audit this tool was not complete. 


3.26 With the implementation of the Immigration and Refugee Protection Act, 
Business Immigration Centres no longer exist. All offices abroad now accept 
and process business applications. There is a need for training in this area, as 
most of the visa officers have little experience with business applications. 
There is no specific course for business applications but they are included as 
part of other courses. Course evaluations that we reviewed recommended 
expanding the section on business immigrants. Business application training 
was included as part of IRPA training and a new module was developed for 
the visa officer course given in the winter of 2003. 


3.27, = Our 2000 Report had concerns about medical inadmissibility. The 
Internal Audit and Disclosures Branch at Citizenship and Immigration 
Canada is planning an audit of designated medical practitioners abroad; 
therefore, we did not follow up on that area. We limited our follow-up mainly 
to observations made in 2000 on medical surveillance in Canada. 


3.28. The Department formed a Medical Services Branch in 2001 that 
unified the management of all areas of the Department that dealt with issues 
of medical admissibility and medical surveillance. 


Excessive demand has been defined 


3.29. We observed in our 2000 Report that the Department had been trying 
for more than 10 years to define the term “excessive demand.” The new 
regulations define the term: an applicant is considered inadmissible if it might 
be reasonably expected that the applicant's health condition would cause 
excessive demand on Canadian health or social services, or if it would add to 
existing waiting lists and would increase the rate of mortality in Canada. 
When determining whether an applicant is likely to create excessive demand, 
the costs of anticipated health and social services for that applicant are 
compared against the average Canadian per capita health and social services 
costs for the next five years. If the applicant is likely to cost these services a 
significant amount for more than 5 years, then the Department will continue 


comparing costs up to 10 years. 


Advice on medical testing and surveillance not yet received 


3.30 Health Canada advises Citizenship and Immigration Canada on which 
diseases should come under mandatory surveillance. In 2001 and 2003 
Health Canada advised the Department on issues related to HIV testing. 

In 2001 Health Canada advised the Department that immigrants should be 
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tested for HIV. At the same time, Health Canada advised the Department 
that it would provide advice soon on whether immigrants with hepatitis B 
require medical surveillance. In April 2002 the Department asked Health 
Canada whether long-staying visitors or visitors who work in close contact 
with Canadians should be tested for HIV. The Department also asked for 
Health Canada’s recommended approach to hepatitis B for immigrants. 
That advice is still outstanding. 


Medical surveillance of immigrants 


3.31 Some immigrants are medically admissible to Canada even though 
their conditions present certain problems. In these cases, departmental 
physicians have found that while the applicants do not pose an immediate 
health threat to Canadians, they need to be kept under medical surveillance 
by the appropriate authorities once in Canada. Immigrants requiring medical 
surveillance must report to public health authorities within 30 days of 
arriving in Canada. Non-active tuberculosis and syphilis are the two diseases 
that require medical surveillance. 


3.32. The Department's main role in the medical surveillance program is to 
provide information to public health authorities on immigrants requiring 
medical surveillance, including their date of arrival in Canada. In our 

2000 Report we noted that we were seriously concerned about an internal 
review reported in October 1999. The review concluded that the Department 
did not know the percentage of immigrants that complies with medical 
surveillance requirements or whether provincial and territorial health 
authorities receive a complete list of immigrants who require medical 
surveillance. 


3.33. The Department has taken several steps to improve medical 
surveillance. It has 


¢ Established a Medical Surveillance Unit. 


¢ Produced an instructional handout for immigrants about medical 
surveillance. The handouts are intended for all immigrants requiring 
medical surveillance. 


¢ Standardized the medical surveillance form. 


* Issued an operations memorandum on medical surveillance roles and 
responsibilities in February 2002 to key players, including those at visa 
offices and ports of entry, to encourage consistent procedures for 
surveillance notification. 


* Worked with the provinces and territories and established a toll-free 
number in each province for immigrants. 


3.34 The need for compliance data. The Department needs to know if 
immigrants requiring medical surveillance are complying with the 
requirement to report to public health authorities. It is working with 
provincial and territorial public health authorities to receive written 
confirmation if and when immigrants report to the authorities. To date, the 
timeliness and methods of written confirmation vary. The Department has 
not yet collected compliance data on the percentage of immigrants that 
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report to the authorities and within what timeframe, but is building a 
database for that purpose. 


3.35 Our field visits indicated that immigration officers at airports were 
familiar with and understood the new instructions. However, some officers at 
land borders were not familiar with them or their importance. After our field 
visits, the Medical Surveillance Unit provided training to immigration officers 
at airports in Vancouver, Toronto, Edmonton, and Calgary and at Ontario 
land borders at Fort Erie, Niagara Falls, and Windsor. 


3.36 If the Department finds out that an immigrant is delinquent in 
reporting for medical surveillance, it takes no action apart from making a 
remark in the Field Operations Support System. The Department denies 
further actions such as citizenship or sponsorship until the immigrant resumes 
medical surveillance. 


Medical surveillance of refugee claimants 


3.37 Lack of medical surveillance for cases of inactive tuberculosis 
among refugee claimants. When refugee claimants arrive at Canadian ports 
of entry, they have not had a medical examination or medical assessment as 
have immigrants. As part of the initial processing of their claims, an 
immigration officer gives them a medical form and list of doctors by province 
and territory. The officer also instructs them to have a medical done by one of 
those doctors within a certain time. This time varies by region. For example, 
in Ontario the time is 60 days; in Quebec it is 5 days. The examining doctor 
sends the medical examination results to Immigration. At that time, 
Immigration does the medical assessment and decides whether medical 
surveillance is needed. The appropriate information is entered into the Field 
Operations Support System. However, Immigration does not notify the 
refugee claimant or the public health authorities in the provinces and 
territories that the claimant requires medical surveillance unless the claimant 


applies for a temporary work or study permit. 


3.38 Refugee claimants are unaware that they have to report to public 
health authorities for medical surveillance if they have inactive tuberculosis. 
They are not under medical surveillance while their cases are in process at 
the Immigration and Refugee Board. This process took, on average, 

10.4 months in 2001-02. If the Immigration and Refugee Board grants 
refugee status, the requirement for medical surveillance is passed on to the 
public health authorities as part of permanent resident status processing. If 
the Board denies refugee status, the public health authorities are not notified 
and yet the failed claimant may remain in Canada for several more months or 
years. 

3.39 Recommendation. Citizenship and Immigration Canada should 
develop a mechanism to inform public health authorities about the 
requirement for medical surveillance involving cases of inactive tuberculosis 


among refugee claimants. 


Department’s response: The Department agrees with the recommendation. 
The Department is in the process of developing a mechanism for the 
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expedient referral of individuals in the refugee determination process who 
require public health surveillance for tuberculosis follow up. The mechanism 
will be implemented in consultation with Health Canada and the Canadian 
Tuberculosis Committee. 


3.40 Under sections 34 to 37 of the Immigration and Refugee Protection Act, 
visa officers can deny applicants entry into Canada on several grounds related 
to criminality and security. 


3.41. Criminality checks consist mainly of examining the information that 
accompanies an application, which includes a police certificate from all 
countries where the applicant has lived for more than six months. Also the 
Department, in consultation with other organizations, has developed profiles 
to help identify individuals who might be associated with organized crime and 
war crimes. When an applicant matches a profile, visa officers can refer the 
case to the Department's Intelligence Branch. The Branch examines the files, 
investigates further, and provides advice to the visa officer responsible for 
processing the application. 


3.42. The Department's offices abroad conduct security checks based on risk 
assessment. The Canadian Security Intelligence Service (CSIS) prepares 
profiles in co-operation with the Department. These profiles identify the 
types of people most likely to be inadmissible for engaging in espionage, 
subversion, or terrorism. Applicants who fit these profiles are referred to CSIS 
for a check. CSIS reports back to Citizenship and Immigration Canada 
whether it has concerns or not. The Service prepares a “no reportable trace” 
report if it has no adverse information on the applicant. The Service prepares 
an inadmissibility brief when it believes that the applicant is inadmissible as 
described in the Immigration and Refugee Protection Act. If CSIS believes that 
the applicant is admissible according to the Act but is or was involved in 
activities described in security provisions of the Act, it prepares an 
information brief. The Service prepares an “incidental letter” if it receives 
information on applicants that could make them inadmissible on matters that 
do not relate to security; for example, health concerns or crimes against 
humanity. CSIS forwards its reports, briefs, and letters to the Intelligence 
Branch, which follows up and advises visa officers. 


3.43 = In our 2000 audit we made several observations and recommendations 
on criminality and security. We were concerned about serious constraints on 
the use of certain information in judicial reviews, limited training, and 10- 
year-old, out-of-date procedural manuals for security reviews. 


Progress in determining inadmissibility 


3.44 The Department has taken steps regarding criminality and security. 
The new Intelligence Branch created in March 2002 defines its role in this 
area as providing information and expertise on intelligence management, 
security, terrorism, organized crime, modern war crimes, and irregular 
migration. The Branch brought together existing intelligence and case 
management resources at headquarters and provided a central point for 
sharing information with partners in the intelligence community. 
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3.45 Other accomplishments include the Royal Canadian Mounted Police 
(RCMP) and CSIS each signing a new memorandum of understanding with 
the Department. The RCMP memorandum covers items such as 
fingerprinting and screening, intelligence, and fraudulent documents. The 
CSIS memorandum describes the sharing of information between the two 
organizations. In addition, the Immigration and Refugee Protection Act contains 
sections that protect security and criminal intelligence that is obtained in 
confidence from foreign or Canadian sources, during admissibility hearings, 
immigration appeals, and judicial reviews. 


3.46 The Department relies on CSIS to take the lead on identifying 
potential security threats. At the offices we visited, we found that security 
profiles were updated with information prepared by CSIS in co-operation 
with the Department. Officers were aware of the security profiles and were 
following departmental procedures for screening applicants. Citizenship and 
Immigration Canada is using the Modern War Crimes System, a new tool to 
help visa officers identify war criminals. However, the manual on security 
screening has not been revised. 


3.47 Submitting documents is an important aspect of the immigration 
process. Documents provide, for example, proof of job experience, language 
proficiency, education, birth, and marriage. Submission of fraudulent 
documents and misrepresentation is a continuing threat to the integrity of the 
Immigration Program. In many countries people can easily obtain fraudulent 
documents and readily use them as support for immigration applications. 


3.48 Verifying the authenticity of local documents and documents from 
other countries requires time and expertise. Our 2000 Report stated that the 
Department had noted an increase in the submission of false statements and 
fraudulent documents. We said the Department had been tolerant of 
applications accompanied by false statements and fraudulent documents. The 
Report recommended that the Department establish and put into effect a 
strategy to reduce the risk that applicants will submit false statements or 
fraudulent documents to an acceptable and manageable level. 


New measures assist in the identification of fraudulent documents 


3.49 In 2002 the Department put into place the following measures: 


¢ Implementation of the Immigration and Refugee Protection Act. The Act 
prohibits certain individuals from re-applying within two years of being 
denied due to misrepresentation. 


¢ Establishment of the Intelligence Branch. The Branch is responsible for 
immigration control officer activities. The Branch also co-ordinates and 
disseminates fraudulent document information. 


- Expansion of the immigration control officer network and development 
of a new work description that includes providing leadership and 
guidance for the identification of fraudulent visa application documents. 
Immigration control officers are now calied migration integrity officers. 
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Quality assurance 


* Revision of the memorandum of understanding between Citizenship and 
Immigration Canada and the RCMP (paragraph 3.45). The 
memorandum, signed in December 2002, clarifies roles and 
responsibilities between the two organizations. It also outlines 
responsibilities for fraudulent travel documents and for investigations 
and referrals prosecuted in Canada. 


These initiatives should help the Department to manage the problem of 
fraudulent documents; however, it is too soon for us to assess their impact on 
departmental operations. 


A need for guidance and to share knowledge 


3.50 We noted that some staff abroad are unclear about their legal right to 
seize documents. There is no policy guidance for officers on seizing 
documents and the procedures for seizing documents vary among offices. 


3.51 Most offices abroad have experience in detecting fraudulent 
application documents. The offices we visited had collections of documents 
and, on their own initiatives, had built separate bases of knowledge. But they 
were not systematically sharing the knowledge among offices that might have 
benefited from it. For example, offices processing a lot of offshore cases had 
no system to access the experience of other offices. We heard-a clear and 
consistent message at all offices we visited that processed offshore 
applications. They said that officers need a way to share knowledge and 
require further training in the detection of fraudulent application documents. 


3.52. In the 2000 Report we recommended that the Department adopt a 
quality assurance framework to ensure consistency in decision-making and in 
the fairness and integrity of the Immigration Program. We observed that the 
quality of decisions was not monitored sufficiently, and that monitoring 
activities varied considerably from one office to another. 


Testing centralized processing 


3.53 In its status reports, the Department identified the “centralized 
processing pilot” as the main way to test its method of quality assurance. One 
objective of the pilot was to test whether it was more efficient to process 
skilled worker applications at one central location or at a network of offices 
abroad. Applications that would have been processed in London, Hong Kong, 
and New Delhi were processed centrally in Ottawa. If officers did not need to 
interview the applicant, the case was finalized in Ottawa. If officers needed to 
interview the applicant or the applicant was refused, the application was 
finalized abroad. A sample of files processed in Ottawa was sent to the 
responsible office abroad for a quality assurance review. The pilot ran from 


December 2000 to August 2001. 


3.54 Another objective of the pilot was to develop and evaluate a method 
for quality assurance for skilled worker applications that could be used at 
offices abroad. The method addressed the following elements: quality of 
decision-making, detection of fraud and misrepresentation, fairness of 
procedures, and efficiency. Each office abroad had a quality assurance officer 
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who reviewed a random sample of files previously processed in Ottawa and a 
random sample of files processed at the office abroad. The method used four 
instruments: file review of initial assessments, interviews with applicants who 
previously had their immigration interviews waived, site visits or telephone 
calls, and file reviews of final decisions. The quality assurance officer also 
reviewed the documents in the file and the visa officer’s use of the 
Department’s information technology systems. 


3.55 The pilot results showed that there were no gains in times or 
productivity from centrally processing skilled worker applications. The results 
also showed that the quality of decisions was lower overall for applications 
processed centrally. The most likely explanation, according to the 
Department, was the lower level of local knowledge at the central processing 
site. The results showed better client service for applications processed in 
Canada. However, those results were based on a very small sample size and it 
was noted that they should be treated with caution. 


3.56 During the pilot the Department tested using digitally imaged files for 
visa applications. Although initial costs to prepare these files were higher 
than paper files, the rest of the file processing showed significant benefits. 


3.57. The Department’s assessment of the pilot suggested that it explore 
processing files at a central location at the beginning and end of the process. 
Visa officers abroad would continue to decide whether to accept or refuse 
applications. It also suggested that this option be considered in the 
development of the Department’s new Global Case Management System. 


Initiatives at offices abroad 


3.58 Although the Department has not decided on the specifics of a quality 
assurance framework, it has been discussed at Immigration Program 
managers’ conferences and initiatives have been taken at some offices abroad. 
For instance, the immigration office in Hong Kong has a quality assurance 
unit of five staff. Its primary task is to examine immigration application 
documents to detect whether they are fraudulent. The Hong Kong office has 
determined that the rate of fraudulent application documents is at least 

25 percent but could be as high as 50 percent. The Department has also 
advised us that the Beijing immigration office has done quality assurance 
reviews on language test results and on economic immigrant applications for 
which an interview had been waived. In addition, the Department has 
created three new anti-fraud positions in offices abroad. 


Immigration is deciding how to proceed with its strategy 


3.59 The Department wrote a draft document, Anti-Fraud/Quality 
Assurance Strategy. It is based on risk management and uses targeted and 
randomly selected application files. The strategy consists of 


- finding and analyzing fraudulent information to improve detection of 


fraud and misrepresentation; 


* providing feedback to the front line; and 
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Offshore processing 


* assessing, through a quality assurance program, the quality and 
consistency of decisions to accept or refuse applicants. 


The strategy discusses the creation of a new position, anti-fraud officer, in its 
offices abroad. This officer would conduct a quality assurance program and 
anti-fraud efforts. For the quality assurance program, the officer would, using 
a random sample of files, review files, conduct interviews and telephone 
verifications, and visit sites. For the anti-fraud program, the officer would 
verify documents with issuers, confer regularly with large document issuers, 
analyze and report on findings, and conduct staff training. The Department is 
deciding how to proceed with quality assurance and anti-fraud efforts. 


The need for a department-wide framework 


3.60 Limited progress in implementing a department-wide framework to 
monitor quality of selection decisions. The Department is developing a 
method and strategy for quality assurance for the economic component but 
has not given specific guidance to the offices abroad, although it has 
requested officers to conduct the anti-fraud work. As a result there is no 
consistency for quality assurance efforts at offices abroad and, at the offices 
we visited, most are doing little. 


3.61. An offshore application is one that an applicant submits to an office 
abroad that does not have responsibility for the applicant's country of 
residence. For example, a resident of Pakistan applies in London, England. At 
the time of our last audit, 48 percent of all economic immigrant applications 
were made offshore. 


3.62 We found several problems with this in our 2000 audit. First, offshore 
applications lowered the quality and consistency of decisions. Second, we 
noted that offshore applications meant a heavier workload for staff in offices 
abroad. The applications took longer to assess and required interviews more 
often. We recommended that the Department take action to ensure that 
applications are processed in the offices that have the skills to make informed 
and consistent decisions efficiently. 


3.63. At the time of our field work the situation remained unchanged. Visa 
officers who process offshore applications at the offices we visited continue to 
find that offshore applications require extra processing time because they 
need to conduct interviews in almost all cases. These applications require a 
significant amount of research on the part of the officers and the Department 
considers them a higher risk for fraud than other applications. Offshore 
applications by economic immigrants represented 26 percent of applications 
in 2001. The decrease from 48 percent at the time of our last audit is due in 
large part to the Department re-classifying some applications from the 
People’s Republic of China. The Department previously had classified them 
as offshore applications. 


Implementing a new regulation 


3.64 The new Immigration and Refugee Protection Regulations contain a 
regulation that would greatly reduce offshore processing when put in place. 
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The regulation requires that applications be made to the Citizenship and 
Immigration Canada office that serves the country where the applicant has 
lawfully resided for at least one year or to the office that serves the applicant’s 
country of nationality. 


3.65 The Department decided to implement the regulation over a transition 
period. It issued a draft operations memorandum on 13 June 2002 to staff that 
all full-service missions would, for the transition period (which was expected 
to be until 1 January 2003), be defined as immigration offices that serve all 
applicants regardless of country of nationality. The Department amended its 
overseas processing manual to state that all visa offices should continue 
accepting offshore applications until directed to do otherwise. As a result, at 
the time of our field work, problems with offshore processing continued. 


3.66 Implementation on 1 May 2003. The Department notified 
immigration offices abroad on 1 April 2003 that the implementation date for 
the regulation is 1 May 2003 and provided them with updated sections of the 
overseas processing manual. After that date applications for permanent 
residence must be submitted according to the regulation. The manual 
describes the regulation and provides guidelines for determining residence 
and nationality. 


3.67. In our 2000 Report we noted that outdated technology in offices 
abroad was a serious obstacle to improving the activities and performance of 
employees. We noted that numerous efforts to upgrade the systems had failed. 
In the meantime, offices abroad were buried in paperwork. It was crucial for 
the Department to identify opportunities for improvement. The Department 
was planning an investment of about $194 million in a new system, the 
Global Case Management System, and was expecting completion of the 
System in five years. We recommended that the Department review its 
systems and practices to find ways to maximize the efficiency of its operations 
and the use of its resources. 


Use of technology has improved 


3.68 The Department has addressed some of the problems outlined in our 
2000 Report by completing several initiatives. These include 


* access by offices abroad to the Field Operations Support System, the 
domestic case-processing system; 

* access by many ports of entry and offices in Canada to the Computer 
Assisted Immigration Processing System, although we noted that some 
immigration officers do not know how to use the system; and 

* access by all visa offices abroad to the Computer Assisted Immigration 
Processing System. 


3.69 The Department continues to develop the Global Case Management 
System to replace the aging systems that Citizenship and Immigration Canada 
has in offices in Canada and abroad. The Department has rated the project as 
high risk due to its size, nature, and complexity. In March 2003 a contractor 
was selected; however, implementation is behind schedule because of delays 


in the selection process. 
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3.70 In 2001-02, the Department collected $451 million in fees for 
processing immigrant and visitor applications and for granting rights for 
permanent residents to land in Canada. The Department estimates that 
nearly 63 percent of these fees, about $284 million, were collected at offices 


abroad. 


3.71. Our 2000 Report noted significant weaknesses in revenue control, 
including the Department’s inability to reconcile accounts and the lack of 
linked computer systems for recording cash receipts and issuing visas. A 
number of visa officers said they lacked the skills to do their job. Internal 
auditors found there was a need for direction from the Department’s 
headquarters on revenue control. We found a lack of systematic monitoring 
of revenue collection activities and a great variation from office to office in 
how rigorously controls were applied. We also noted the need for an 
appropriate framework for, and management of, revenue control and more 
internal audits. 


3.72 Citizenship and Immigration Canada and the Department of Foreign 
Affairs and International Trade (DFAIT) share responsibilities for 
immigration revenue control at missions abroad. Generally, Citizenship and 
Immigration Canada staff collect fees from applicants, issue receipts, and 
record the fees in the cash receipt system. Immigration then turns the funds 
and a summary over to DFAIT for accounting purposes and deposit in a 
financial institution. In some locations applicants pay fees at a local financial 
institution. In these cases, immigration staff record the bank receipts in the 
cash receipt system and turn the summary over to DFAIT for accounting 
purposes. Immigration cost recovery officers oversee Immigration’s 
procedures. Immigration establishes policies and procedures for its 
responsibilities and gives direction to cost recovery officers. A 1992 
memorandum of understanding between the two departments indicates that 
DFAIT is responsible for immigration cost recovery functions; more 
specifically, for providing accounting services, including cash receipt and 
deposit facilities. 


The Department has made progress in revenue control 


3.73. Since the 2000 Report, the Department has continued to implement a 
cash receipt system which is now installed in 71 missions. The missions we 
visited all had the latest version of the software and guide for the system. 


374 The cash receipt system is not linked to the Computer Assisted 
Immigration Processing System, the system used to process immigration 
applications and issue visas. The Department cannot reconcile accounts; that 
is, verify that the number of files processed and visas issued corresponds to 
the fees collected. The Department has advised us that it will build this ability 
into the new Global Case Management System. In addition, in July 2001 the 
Department and DFAIT issued a joint message to missions on cost recovery 
outlining current procedures and new procedures to improve controls. 
Because cash receipt and accounting for it are areas of high risk and 
vulnerable to manipulation, it is important that the departments follow up to 
ensure that these procedures are followed. 
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3.15 The examination of other banking options by Immigration and DEAIT 
is ongoing. The Department is working on guidelines that missions can use 
when researching alternate banking arrangements. However, these are not 
complete and were not included in the new draft cost recovery manual that 
the Department is working on. There are 27 missions with banking 
arrangements that enable applicants to pay immigration fees at a financial 
institution. 


3.76 The Department, DFAIT, and other partners began pilot projects in 
two locations abroad using credit cards to pay for some immigration fees. 
These pilots were scheduled to end in April and July, 2003. Departmental 
comments indicate that the pilots were not successful and the departments 
are looking at other payment options that would still use credit cards. 


Tools and training are still a concern 


3.77 While the Department is working on a new cost recovery manual for 
missions abroad, the new manual is about two years behind schedule. The 
offices we visited had different versions of a cost recovery manual with the 
exception of one office which had no manual. 


3.78 The Department has added modules on risk management, 
malfeasance, and cost recovery to the Immigration Program manager's 
course. It has also provided tools to assess program integrity and identify risks 
and a kit to use on arrival at missions with which to conduct a cost recovery 
audit. The basic visa officer’s course includes a module on cost recovery and, 
in the winter of 2003, a module on the cash receipt system was added. 
Modules on ethics and malfeasance have been added to courses for local staff. 
The Department of Foreign Affairs and International Trade has developed 
and delivered a fraud awareness course at missions. However we are still 
concerned about the training of officers at some offices abroad. Cost recovery 
officers at some of the offices we visited said they have not had sufficient 
training in revenue matters and would appreciate that training and an up-to- 
date cost recovery manual. 


3.79 Visa forms control is an essential measure to guard against abuse. In the 
2000 Report we had several concerns. We found that the Immigration Visa 
and Record of Landing document (IMM1000) was outdated and easy to 
falsify. In addition, departmental records showed that some visas had been 
stolen and that procedures were not in place to ensure that officers 
consistently applied controls. 


The new Permanent Resident Card 


3.80 The Department replaced the IMM1000 as of 28 June 2002. A new 
form, the Confirmation of Permanent Residence (IMM5292) confirms the 
residence status of new immigrants. It is used to issue the new Permanent 
Resident Card. This card has a number of security features that the IMM1000 
did not have. Another new form, the Permanent Resident Visa (IMM1346), 


is included in an individual’s passport as a travel document. 
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Problems still exist with forms control 


3.81 We found several problems with forms control during our visits to 
immigration offices abroad. Officers were not always following the required 
control procedures. Three examples: some offices were not reporting 
quarterly inventories of documents to headquarters; one office was not 
reconciling on a daily basis visas that were used; and not all offices had two 
Canadian officers present when counting inventories of forms. 


3.82 One Canada-based officer is selected at each office abroad to be a 
forms control officer. Most forms control officers at the offices we visited felt 
that they needed training and guidance for their position. To address this 
need, the Department has developed new instructions on forms control and 
these are available electronically. 


3.83 We were concerned in 2000 that the resources allocated to internal 
audit activities were cut significantly during a reorganization of the 
Department, and again as a result of Program Review. We recommended 
strengthening the role of internal audit throughout the organization. This has 
now been done. The Department established the Internal Audit and 
Disclosures Branch in 2002, hired a director general, and assigned additional 
resources. 


3.84 In our 2000 Report we were concerned that the Department’s 
performance indicators and its reports to Parliament on skilled workers and 
business immigrants were focussed primarily on meeting immigration levels. 
We also noted that the Department needed better information on the use of 
resources and on the quality and uniformity of selection and admissibility 
decisions. 


Information for Parliament is incomplete 


3.85 Performance indicators and reports to Parliament remain focussed 
on immigration levels. The Department does have other information on the 
economic component readily available that could be reported to Parliament; 
for example, number of applications on hand and processing times for 
applications. In our view this is important information that needs to be 
reported to Parliament. 


3.86 The Department reports annually to Parliament in its departmental 
performance report. It also presents an annual report to Parliament on the 
operation of the Immigration and Refugee Protection Act for the preceding 
calendar year as required by section 94 of the Act. We reviewed several years’ 
worth of departmental performance reports and the 2002 Annual Report to 
Parliament on Immigration. We are concerned that the information 
presented to Parliament on the number of skilled workers entering Canada 
may be misunderstood. For example, the 2002 Departmental Performance 
Report states that the number of skilled workers landed in Canada in 2001 is 
137,119. The information would be more clearly presented as 137,119 skilled 
workers, consisting of 58,860 principal applicants and 78,259 dependants. 
The 2002 Annual Report to Parliament on Immigration also did not provide 
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the composition of skilled workers. We have the same concern for the 
presentation of the numbers of business immigrants. We noted that the 
information does exist in a document, Facts and Figures 2001, which can be 
found on the Department’s Web site. www.cic.gc.ca/english/pub/facts2001/ 


index.html 


3.87. Recommendation. The Department should include additional 
information in its departmental performance report, such as the number of 
applications on hand and processing times for applications. In that report and 
in its annual report to Parliament on immigration, the Department should 
give the numbers of both principal applicants and accompanying family 
members. 


Department’s response: The Department agrees. In its departmental 
performance report and its annual report to Parliament on immigration, the 
Department will include additional information regarding the composition of 
the skilled worker and business immigrant classes, as well as inventory levels 


and processing times. 


Conclusion 


3.88 The Department has acted on many of the observations and 
recommendations made in our April 2000 Report, Chapter 3, The Economic 
Component of the Canadian Immigration Program. Many of the 
recommendations that we made in that Report have been addressed in the 
new Immigration and Refugee Protection Act and its regulations. It is too soon 
after the implementation of IRPA for us to reach a conclusion on the impact 
of the new criteria and tools for selecting skilled worker and business 
immigrants. The new fraudulent document initiatives should help address 
problems noted in our 2000 audit. 


3.89 Greater attention needs to be paid to medical surveillance of 
immigrants and refugee claimants to ensure that public health authorities are 
notified promptly of individuals that require surveillance. The Department 
still has to put into place a department-wide quality assurance framework to 
ensure the consistency, fairness, and integrity of selection decisions. 
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About the Follow-up 


Objective 


The objective of this follow-up was to assess the extent of Citizenship and Immigration Canada’s actions to address 
recommendations in our April 2000 audit. 


Scope and Approach 


This follow-up focussed on the recommendations made in our 2000 Report, Chapter 3, The Economic Component 
of the Canadian Immigration Program. We also looked at medical surveillance of refugee claimants. We did not look 
at controls over medical admissibility at offices abroad. We reviewed two status reports by the Department on the 
action it has taken in response to our recommendations. We interviewed staff at Citizenship and Immigration 
Canada, the Canadian Security Intelligence Service, and the Royal Canadian Mounted Police. We met with 
immigration staff at national headquarters and in five regions in Canada. We visited nine offices in other countries: 
Accra, Guatemala City, Hong Kong, London, Manila, New Delhi, New York, Singapore, and Vienna. We visited 

14 ports of entry including airports, land border crossings, and marine ports. We interviewed staff, reviewed files and 
management reports, and analyzed data. 


Criteria 


We expected that Citizenship and Immigration Canada would have made satisfactory progress in putting in place our 
recommendations. 


The criteria from the 2000 audit remain relevant. Therefore we expected to find the following: 

¢ roles and responsibilities within the federal government that would facilitate efficient and effective 
management of the Canadian Immigration Program; 

* organization of activities and allocation of resources to take into account existing risks and to encourage 
reaching immigration levels in an economical and efficient way while maintaining the Program’s integrity; 

* objectives of the economic component to be clearly set out and the selection criteria helpful in meeting those 
objectives; 

* mechanisms in place to ensure the quality of decisions and the Program’s integrity; 

* acontrol framework to protect the main assets; 

* management practices to make it possible to monitor the Department’s performance carefully; and 


* information intended for Parliament to be relevant, reliable, and complete. 
Audit Team 


Assistant Auditor General: Shahid Minto 
Principal: John Hitchinson 
Directors: Janet Jones 
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Appendix Selection criteria for immigrants in the economic component 


Section 12 of the /mmigration and Refugee Protection Act provides that foreign nationals may be selected as members of 
the economic component based on their ability to become economically established in Canada. The economic component 
is made up mainly of two streams: skilled workers and business immigrants. The Minister decides and makes available to 
the public the minimum number of points required for the establishment of the economic applicant in Canada. The 
province of Quebec sets its own selection criteria for immigrants under the economic component. 


Skilled workers 


Initially, skilled workers are assessed against certain minimal employment experience requirements. If these requirements 
are not met, the application is refused. Otherwise they are assessed against the following selection criteria and must 
receive a minimum of 75 points. 


Education 25 
Official language proficiency 24 a 
Experience 21 
Age 10 
Arranged employment ; 10 
Adaptability 10 


Skilled workers must also have a certain amount of available funds or arranged employment. 


Business immigrants 


Business immigrants are selected to support the development of a strong and prosperous Canadian economy in which the 
benefits of immigration are shared across all regions in Canada. Members of the business immigrant class include 
investors, entrepreneurs, and the self-employed. During the selection process, officers first assess whether the applicant 
meets the definition of investor, entrepreneur, or self-employed. If not met, the application is refused. Otherwise officers 
assess the applicant using a point system based on selection criteria that measure the applicant's capacity to get 
established successfully. 


Investors 


The definition states that investors must demonstrate business experience, which includes two options: 
* management and control of a qualifying business of sufficient size; or 
* management of a business, or a portion of a business (without the need to own it), with at least five employees. 


Investors must also have a legally obtained minimum net worth of $800,000 and make an investment of $400,000. 
Investors must deposit CAN$400,000 with the Receiver General for Canada before a visa can be issued. Subsequently, 
the funds will be distributed to the provincial funds for investment. After five years, the provincial funds will repay 
Citizenship and Immigration Canada for subsequent payment to the investors. The participating provinces will continue to 
be responsible for guaranteeing the payment of their respective shares. 


Entrepreneurs 


Entrepreneurs must demonstrate business experience consisting of management and conirol of a qualifying business of 
sufficient size. Entrepreneurs will be required to document that, for a period of at least one year, within a period of not 
more than three years after landing, they owned and managed a qualifying Canadian business that meets any two of the 
defined requirements for jobs, sales, net income, and equity. 


Report of the Auditor General of Canada—May 2003 Chapter 3 i] 21 


CITIZENSHIP AND IMMIGRATION CANADA—THE ECONOMIC COMPONENT OF THE CANADIAN IMMIGRATION PROGRAM 


Self-employed 


Self-employed people must have the intention and ability to create their own employment and make a significant 

contribution to the cultural or athletic life of Canada, or to create their own employment by purchasing and managing a 
farm in Canada. Immigrants who have been self-employed in cultural activities or in athletics, or have participated at a 
world class level in cultural activities or in athletics, or have farm management experience are eligible within the class. 


Business immigrants will be evaluated against the following grid and must receive a minimum of 35 points: 


Business experience (investors or entrepreneurs), or | 315} 
relevant experience (self-employed) 


Age 10 
Education | 25 
Official language proficiency 24 
Adaptability 6 


Source: Citizenship and Immigration Canada 
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All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 
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Correctional Service Canada 
Reintegration of Male Offenders 


Main Points 


41 Correctional Service Canada has substantially implemented many of 
the recommendations we made in our 1999 audit. It has made more time 
available for offenders to participate in needed programs while incarcerated. 
It has improved the quality of the reports it prepares for the National Parole 
Board to decide whether an offender should be released conditionally into the 
community. In the community, the Service has made a significant 
improvement in meeting its standards for frequency of contact with offenders 
on parole. Senior management commitment and leadership have been 
important in achieving these results. 


4.2 In two areas that are critical to reintegrating offenders effectively, 
however, Correctional Service has been slow to address our recommendations 
of previous audits. Not enough programs are available to offenders in the 
community to meet the need. The Service has made little progress in 
implementing key aspects of its approved national strategy for providing 
offenders with appropriate employment programs. 


43 We found three other important areas that need attention where we 
did not make a recommendation in 1999: 


¢ The Service has not fully tested the inter-rater reliability of the tools it 
uses to assess an offender’s needs and the risk he presents. 


* Parole officers are critical to the successful reintegration of offenders, 
but the level and quality of their training is weak. 


* Turnover among parole officers appears high, with a potential impact on 
offender reintegration operations. 


44 Since 1994, we have conducted five audits that examined various 
practices for reintegrating male offenders into the community, a vital part of 
the Service’s mandate. In our two most recent audits we have reviewed the 
same areas to determine how much progress the Service has made on 
addressing our previous recommendations. The audit reported here looks 
specifically at how Correctional Service has acted on our 

1999 recommendations. 


45 In 1999 we recommended that the Service improve its timeliness in 
acquiring official documents needed to assess offenders when they enter 
institutions. We said it needed to be more efficient in preparing casework to 
meet the offender’s first date of eligibility for parole. It needed to clarify and 
implement a strategy for employment programs geared to reintegrating 
offenders. And we said it should improve the quality of its reports that the 
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National Parole Board uses to grant the conditional release of offenders into 
the community. We also noted the need for better adherence to national 
standards for frequency of contact with offenders in the community. 


Correctional Service Canada has responded. Correctional Service has 
generally accepted our recommendations. Its responses indicate what it is 
doing or plans to do to address them. 
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Introduction 


46 Correctional Service Canada has two main responsibilities: the 
incarceration of offenders and their safe reintegration into the community. 
The Service spent $492 million on reintegration activities in 2001-02— 
about 33 percent of its total spending, roughly the same proportion it spent 
on reintegration in 1997-98, 


47 The Corrections and Conditional Release Act says that the purpose of 
conditional release is to contribute to public safety by releasing offenders at a 
time and in a manner that increases their chances of successfully 
reintegrating into the community. The Act gives the National Parole Board 
authority to grant the conditional release of offenders on full parole when 
they have served one third of their sentence. Offenders become eligible for 
day parole six months before the date of their eligibility for full parole 
(Exhibit 4.1). Under Accelerated Parole Review, offenders who are serving 
their first federal sentence and were not convicted of a violent crime or 
serious drug offence can be released on day parole after serving one sixth of 
their sentence, unless there are reasonable grounds to believe they will 
commit a violent offence before their sentence ends. 


4.8 After serving two thirds of their sentence, most offenders who have not 
been paroled are entitled by law to be released into the community (statutory 
release), where their supervision continues until the sentence expires. In a 
small number of cases, Correctional Service will recommend to the National 
Parole Board that an offender be kept in prison until the end of his sentence. 
The Board detains an offender when it is likely that, if released before his 
sentence ended, he would commit an offence involving death or serious 
harm, a sexual offence against a child, or a serious drug offence. 


49 Most offenders under supervision in the community are subject to 
conditions such as refraining from the use of alcohol and from associating 
with criminals. If an offender breaches any of these conditions and the 


Exhibit 4.1 Milestones in a fixed-term sentence of six years 


Offender is 1/3 of 2/3 of Sentence 
sentenced to 6 years sentence sentence expires 
& (2 years)», (4 years) (6 years) 
<" S 
= ™, 


EE 
| 
Accelerated months 
Parole Review 
(12 months) Statutory release or 
Day parole continued detention 
eligibility 
(18 months) 
Full parole 
eligibility 


Source: Corrections and Conditional Release Act 
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supervising parole officer feels that the risk to the community has been raised 
to an unacceptable level, the National Parole Board may revoke the 
offender’s parole and have him returned to prison. 


The process for reintegrating male offenders 


410 Correctional Service uses a case management process to manage the 
reintegration of offenders. The process has a number of stages that include 
the following: 


* obtaining official documents required for assessing the security risk and 
the needs of each offender; 


* assessing the offender to identify the factors that led to his criminal 
behaviour and developing a correctional plan to address those factors; 


* assigning and scheduling the offender to participate in reintegration 
programs and other interventions to prepare him for release hearings 
before the National Parole Board; 


* assessing whether participating in programs and other interventions has 
helped reduce the risk that the offender will commit another offence 
after being released; 


* making recommendations to the National Parole Board on the 
offender’s suitability for conditional release into the community; 


¢ releasing the offender—on parole by the National Parole Board or, by 
law, after two thirds of the sentence (statutory release), or at the end of 
the sentence (warrant expiry); 


* providing supervision, further programs, and assessment in the 
community until the end of the sentence; and 


* reporting to the National Parole Board if an offender in the community 
presents an elevated risk that warrants a review by the Board. 


Trends in the offender population 


411. The number of male offenders in federal correctional institutions and 
under supervision in the community has declined over the last few years, 
while the split between the two groups has remained fairly constant at 

60 percent and 40 percent respectively (Appendix). A total of about 

500 fewer offenders were admitted to federal prison in 2001-02 than in 
1997-98, and the total number of releases declined by about 1,000 during the 
same period. As we reported in 1999, offenders continue to be released at a 
later point in their sentences—as illustrated by the growing proportion of 
releases that are statutory releases. 


4.12 National Parole Board data covering the period 1994-95 to 2001-02 
indicate that the conviction rate for violent offences by offenders supervised 
in the community appears to have declined. Exhibit 4.2 outlines this decline 
among offenders on day parole, full parole, and statutory release. 
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Exhibit 4.2 Conviction rates for violent offences by supervised offenders* 


per 1,000 offenders 


== Statutory release 
ees (Jay parole 
exe Full parole 


1994-95 1995-96 1996-97 1997-98 1998-99 1999-00 2000-01 2001-02 


*Supervised offenders includes offenders on parole or statutory release, temporarily detained in a federal 
penitentiary, or unlawfully at large. 


Source: National Parole Board Performance Monitoring Report 2001-2002 — July 2002 


Focus of the follow-up 


413 The purpose of this follow-up audit was to assess the extent to which 
Correctional Service Canada has acted on recommendations made in 
Chapter | of our 1999 Report. In 1994 and 1996 we had reviewed all aspects 
of the reintegration process, from the admission of offenders into federal 
institutions to their supervision in the community after release. 


414 We reported in 1999 that while the Service had made a concerted 
effort to respond to previous observations, it still needed to make the 
following improvements: 


* more timely acquisition of official documents for initial assessment of 
offenders; 


* more timely casework preparation to meet the offender’s first parole 
date; 


* aclear operational strategy for offender employment programs; 


* better quality of reports to the National Parole Board to grant the 
conditional release of offenders into the community; and 


* improved adherence to national standards for frequency of contact with 
offenders in the community. 


4.15 Some quantitative information in this chapter is based on data 
provided by Correctional Service. We have attempted to assess these data 
through a process of analysis, comparison, and discussion. Unless otherwise 
indicated in this chapter, these data should be treated as unaudited. 
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Assessing offenders at intake 
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416 Further details on our audit objective and criteria are presented in the 
About the Follow-up at the end of the chapter. As in our previous audits, this 
audit did not include issues specific to Aboriginal offenders. 


Observations and Recommendations 


417 The first step in managing an offender’s case is the intake assessment 
process, which determines the appropriate level of security for the offender. A 
correctional plan is also prepared, outlining the factors that led to the 
offender's criminal behaviour and the programs he will need to address them. 
The Service must complete the correctional plan as soon as possible after the 
offender enters federal custody; late completion leaves little time for offenders 
serving short terms to participate in the programs they need to become 
eligible for release. 


Inter-rater reliability of assessment tools needs more testing 


4.18 The Custody Rating Scale is the assessment tool used to determine 
what level of security an incoming offender warrants. If an instrument has a 
high inter-rater reliability, different parole officers assessing the same offender 
will likely assign him to the same security level. If its reliability is relatively 
low, different parole officers will probably assign the same offender to different 
security levels. Assigning an offender to a higher level than necessary can 
reduce opportunities for him to participate in reintegration programs. It also 
increases the cost of incarceration. 


419 In 1999, we found that Correctional Service had adequately verified 
that the Custody Rating Scale was scientifically valid for assessing male 
offenders; however, it had not thoroughly tested its inter-rater reliability. The 
use of an unreliable instrument could call into question the testing of its 
validity. 


4.20 As part of its ongoing research activities, the Service recently 
undertook a preliminary test of the Custody Rating Scale’s inter-rater 
reliability among a small number of parole officers. Initial results showed that 
about 25 percent of the officers assigned higher security levels to the same 
offender. Given the potential impact of a problem with inter-rater reliability, 
these preliminary test results are a concern and reaffirm that the Service 
needs to do further work. 


4.21. Our follow-up found that Correctional Service has yet to complete a 
full test of its Custody Rating Scale for its inter-rater reliability in classifying 
male offenders. Periodic ongoing tests of inter-rater reliability are required to 
assess the adequacy of the way the tool is applied, the current training given 
to those using it, and controls on the quality of its results. 


4.22 Recommendation. Correctional Service Canada should fully test the 
inter-rater reliability of the tools it uses to assess male offenders at intake and 
should take the necessary action indicated by test results. 
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Correctional Service’s response. Correctional Service Canada’s research 
confirms the predictive validity of the assessment tools it uses. The Service 
accepts the suggestion that additional research should be done on inter-rater 
reliability and this is included in the 2003-2004 Research Plan. 


Timely preparation of offenders’ correctional plans is still a challenge 


4.23 In the past, our Office has examined the completion of the offender’s 
correctional plan from two perspectives: timely acquisition of documents 
required to assess the offender and the amount of time needed to analyze this 
information and complete the correctional plan. 


4.24 The Service must gather information on the offender and the offence 
from its own records and from outside agencies. Its policy requires that it have 
three documents available to assess the offender’s risk and needs: the 
offender’s criminal history (for example, police reports); an official version of 
the offence (for example, the judge’s comments); and the Post-Sentence 
Community Assessment (prepared by community parole officers). All official 
documents are needed far enough in advance to prepare the offender’s 
correctional plan on time. 


425 In 1999 we noted that while the Service had reduced some of the 
delays in acquiring these documents, in about one quarter of the cases it did 
not meet its own standards for timeliness. We recommended that the Service 
find ways to further reduce the delays in getting the necessary information on 
offenders. Our follow-up found that Correctional Service’s current data show 
a further reduction in the average number of days it takes to acquire official 
information on offenders (Exhibit 4.3). 


Exhibit 4.3 Average number of days to receive police reports, judge’s comments, and post-sentence 
community assessments (1999-2002) 


os Average number of days : 
oy | = —o a ee Post-sentence — 
Time period =| Police reports _ | Judge's comments | community assessments ‘ 
1999-00 | Bass 34 30 
2000-01 Zo 35 27 
2001-02 2 eu 23 


4.26 As of April 2001, Correctional Service standards require that the 
correctional plan be completed within 70 days of the admission date for 
offenders serving less than four years and within 90 days for the rest. The 
Service still has difficulty meeting its standards for timely completion of 
correctional plans (Exhibit 4.4). In 2001-02, only about 70 percent of all 
correctional plans were completed on time. The Service has reviewed its 
completion rate for the current year to date and found that nationally, it has 


improved to 84 percent. 
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Case management 


Exhibit 4.4 Percentage of correctional plans completed on time (April 1999-December 2002) 


Percentage 


100 


1999-2000 2000-01 2001-02 2002-03 


By quarter-year 
Source: Correctional Service Canada 


4.21. More timely acquisition of required official documents represents an 
improvement in one aspect of completing correctional plans on time. 
However, a study of the offender intake process conducted for the Service in 
2002 concluded that better scheduling and assignment of resources may be 
necessary to complete all correctional plans on time as required. 


4.28 Recommendation. Correctional Service Canada should identify the 
measures needed to meet its existing standards for on-time completion of 
correctional plans and should take the necessary corrective action. 


Correctional Service’s response. We have focussed our attention on this 
area and have already seen improvement. We will continue to monitor on a 
monthly basis and take appropriate action as required. 


There is more time for offenders to complete programs before their first parole 
eligibility date 


4.29 In 1999 we recommended that Correctional Service Canada improve 
the efficiency and timeliness of its case management process and the 
preparation of its reports on offenders to meet its approved standards. 


430 We had estimated in 1996 that, on average, three rehabilitation 
programs were prescribed for low-risk/low-need offenders, and the shortest 
possible time in which they could complete the three programs was 105 days. 
We calculated that offenders serving a two- to three-year sentence had only 
19 days to complete the programs before their earliest date of eligibility for 


parole. By 1998, this had improved to 60 days, and in 2001-02 to 79 days. 


4.31 These results suggest that most offenders who require rehabilitation 
programs are now more likely to complete them on time. Further, some 
offenders start critical programs in the Intake Unit while they are still 
awaiting placement in an institution. More timely completion of all 
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correctional plans would allow added time for reintegration programs, and 
this would help offenders meet their parole eligibility dates. 


Some offenders remain incarcerated after they become eligible for parole 


4.32 Parole hearings can be delayed for several reasons—for example, if an 
offender feels that the National Parole Board is unlikely to release him, if he 
has a court case or an appeal pending, or if Correctional Service is unable to 
provide the required programs in time to prepare the offender for release. The 
National Parole Board can also adjourn a hearing if it does not have the 
required documentation in time for members to prepare for the hearing. 


4.33 Exhibit 4.5 shows that from 1998 to 2002, the numbers of annual 
withdrawals, postponements, and administrative adjournments of hearings 
(in combination with rescheduled hearings) have been fairly stable. However, 
waivers (an offender’s waiving of his legal right to a hearing) have increased 
somewhat. In 2001-02 almost 9,300 of the possible 36,400 decisions on the 
release of offenders were delayed. 


434 In 1999 we recommended that Correctional Service Canada regularly 
analyze the reasons for the number of offenders who remain incarcerated 
beyond their first date of eligibility for parole. That year, Correctional Service 
conducted a national study of 1,300 offenders still incarcerated after their 
date of eligibility for full parole in order to examine the validity of the waivers, 
postponements, and withdrawals of parole hearings. 


4.35 The study indicated that in most cases, the reasons for the delays were 
beyond Correctional Service’s control—for example, release was not 
recommended; the offender had refused programs; or offenders serving short 
terms did not have time to complete the programs prescribed in their 
correctional plans. Correctional Service did determine that seven percent of 
the delays were caused by factors within its control, such as insufficient 
program capacity or problems with scheduling. The Service concluded from 


Exhibit 4.5 Waivers, withdrawals, postponements, and adjournments (1998-2002) 
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Source: National Parole Board 
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its study that not many good candidates for release were being detained 
unnecessarily. National and regional monitoring will continue, with periodic 
sampling of offenders who are past their date of eligibility for full parole. 


436 Despite the improvement that the study showed, almost 100 offenders 
still were not prepared for release in time for their parole eligibility dates. The 
Corrections and Conditional Release Act requires that the Service maintain 
offenders at the lowest level of custody possible, including supervision in the 
community. Delayed releases have cost implications, since it is more 
expensive to keep offenders in prison than in the community. However, 
protecting society must always remain the paramount concern. 


4.37 Recommendation. Correctional Service Canada should eliminate all 
delays within its control that affect the timely preparation of offenders for 
their first dates of eligibility for parole. 


Correctional Service’s response. As acknowledged in the chapter, there are 
very few offenders who are good candidates for release who see their parole 
review delayed due to reasons within the Service's control. It is also important 
to recognize that the overriding principle in federal corrections legislation is 
the protection of the public, and therefore our work in preparing offenders for 
their parole review is grounded on that principle. The Service recognizes the 
importance of timely preparation of offenders and will continue to examine 
the reasons within its control for delays and take appropriate action. 


Senior correctional officers are still not fulfilling their case management 
responsibilities 


4.38 Managing offenders’ cases in the institutions requires teamwork 
between senior correctional officers (who also have security duties) and 
parole officers. Their collaboration and exchange of information are crucial to 
understanding changes in offenders’ behaviour—information that parole 
officers use in preparing reports for offenders’ hearings before the National 
Parole Board. 


439 In 1996 and 1999, we observed that senior correctional officers were 
not consistently carrying out the case management duties as required by 
Correctional Service policy. Among other things, these officers were expected 
to complete clear, concise reports for case management purposes, inform 
colleagues about significant incidents and behavioural changes of inmates, 
and participate in assessing inmates. We recommended that these officers 
perform the case management duties required of them by policy. 


4.40 Recently, Correctional Service adjusted its division of case 
management responsibilities. As a result, many of the senior correctional 
officers’ responsibilities for case management reporting were shifted to the 
institutional parole officers. While the Service made this operational decision 
over a year ago, it has just started the process of rewriting and re-evaluating 
the job description for senior correctional officers. 


441. The new division of responsibilities still requires that senior 
correctional officers observe changes in offenders’ behaviour. Their case 
management responsibilities now are mainly to record monthly the offender’s 
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behaviour and progress against his correctional plan objectives; after a 
successful temporary release of an offender into the community, to submit 
reports for subsequent releases; and to recommend private family visits. 


4.42 An internal audit of case management in 2001 examined, among other 
things, one function that senior correctional officers are still required to do— 
recording offenders’ progress and behaviour monthly, on time, and at the 
required standard of quality. The content of most of the reports reviewed by 
the internal auditors did not meet the Service’s requirements. 


4.43 Recommendation. Correctional Service Canada should complete the 
rewriting and re-evaluation of the job description for senior correctional 
officers and ensure that they fulfil the case management duties still assigned 
to them, as reflected in the approved division of case management 
responsibilities. 


Correctional Service’s response. Correctional Service has a commitment to 


review and update the senior correctional officer (CX02) work description by 
September 2003. 


Quality of reports to National Parole Board for release decisions has improved 


4.44 In deciding whether or not to grant the conditional release of offenders 
from prison, the National Parole Board examines reports and file information 
pertaining to the offender. Among other things, it places a lot of importance 
on the quality of the parole officer’s analysis in reports from Correctional 
Service. In 1999, we concluded that the quality of these reports was a 
concern. We recommended that Correctional Service address known 
deficiencies in the quality of its reports to the National Parole Board. 


4.45 In this follow-up audit, we used the same methodology to determine to 
what extent the Correctional Service reports help Board members make 
informed decisions on the conditional release of offenders. We selected a 
random sample of reports sent to the National Parole Board for release 
decisions from | January to 30 June, 2002. We examined whether the reports 
had enough information so that a Board member could base a decision solely 
on their content (good reports); whether they had some gaps in information 
or analysis that would require additional review of the offender's file 
(adequate reports); or whether they were of so little value or so unclear that a 
decision would require a full review of the file (poor reports). We also selected 
a sample of reports on offenders who presented a higher risk and had greater 
needs when the decision on their release was before the National Parole 
Board. We tested this sample separately to determine whether the quality of 
these reports differed. 


4.46 We found that the quality of reports to the National Parole Board has 
improved since our 1999 audit (see Exhibit 4.6). In 1999, we rated 11 percent 
of all reports sampled as poor in quality. In 2002, 6 percent were rated poor. 
However, one in eight files on the higher-risk offenders contained reports that 
did not provide sufficient information to Board members. We suggest that the 
Service continue to monitor this area to maintain the current level of 
improvement and strive to achieve better quality in its reports. 
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Programs for offenders 
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Exhibit 4.6 Improved quality of reports to the National Parole Board on offenders 


Good 


Adequate 


Poor 


447 An internal audit review in 2001 of the case management process also 
examined the reports sent to the National Parole Board for decisions on 
releasing offenders. The review focussed on compliance with Correctional 
Service’s policy and procedural requirements for the case management of 
offenders serving short terms. The review indicated that the reports did not 
meet the Service’s current requirement to include an assessment of the 
offender’s progress against his correctional plan, the impact of programs 
taken, and changes in the offender’s attitude and his adjustment to 
incarceration. 


4.48 One of the problems we found in 1999 was inconsistent control over 
the quality of these reports to the National Parole Board. Unit managers (who 
supervise teams of correctional and parole officers) were responsible for 
quality control but could not fulfil that role, partly because they lacked 
enough background and training in case management. 


4.49 During our recent field visits, we found that the level of Correctional 
Service’s quality control over reports to the National Parole Board was still 
uneven. The Service told us it is developing a training course for unit 
managers that focusses on quality oversight and feedback. 


4.50 The Service has a wide range of rehabilitation programs for offenders 
to reduce the risk that they will reoffend. These programs can be divided into 
two broad categories: intervention and employment. Intervention programs 
have been developed to address the offenders’ characteristics that have led to 
their criminal behaviour. They deal with such areas as substance abuse, family 
violence, and sex offences. 


4.51 Employment programs are designed to make an offender more 
employable when he is released. They include vocational training, prison 
industries, and institutional work in areas such as the kitchen, laundry, and 
maintenance. 


4.52 In 1996 we concluded that both intervention and employment 
programs lacked a management framework that would enable senior 
management to assess these programs and strategically reallocate funds to 
those that work best. 
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453 In 1999, we reported that Correctional Service had a framework in 
place for evaluating the effectiveness of intervention programs. It had begun 
setting up panels of internationally recognized experts, who provided 
accreditation of a few national and local programs. Since then, all but three of 
38 core programs for offenders in the institutions have been accredited. The 
Service has also made a concerted effort to standardize the array of available 
programs by replacing non-accredited local programs with accredited 
national programs. 


Implementation of the strategy for employment programs is just beginning 


4.54 Research shows that the risk of reoffending is much higher among 
offenders whose employment patterns have been unstable. To help offenders 
become more employable, the Service provides a series of employment 
programs and job training. 


455 In both 1996 and 1999, we recommended that Correctional Service 
develop and implement a coherent strategy for providing employment 
programs. After six years, it has a strategy but has only just begun to 
implement it. 


4.56 In April 2000, the Service consolidated all employment programs for 
offenders under the responsibility of Corcan, a special operating agency 
within the Service. In December 2000, Corcan formalized its employment 
strategy in the Employment and Employability Program. It redefines the way 
Correctional Service values and assigns work and vocational placements. The 
success of this employment strategy requires the involvement and 
co-operation of the Service’s regional and institutional operations with 
Corcan. 


4.57 The goal of the Employment and Employability Program is to ensure 
that offenders are ready for the job when they are released into the 
community. Its purpose is to give offenders the chance to develop 
employment skills, gain certified work experience, and understand what is 
expected in private-sector employment—for example, the pace, the quality of 
work, and the hours of work. 


458 Under this strategy, employment is now viewed as a core rehabilitation 
program like any intervention program. As such, it requires 


* assessing to what extent a lack of employment skills has been a critical 
factor in the offender’s criminal behaviour; 


* ensuring that employment programs are designed to remedy these 
deficiencies; 


* placing the offender in an appropriate employment program; 
* evaluating the offender’s progress; and 
* providing follow-up support in the community. 


459 Assessing the offender’s needs. During intake assessment, 
Correctional Service traditionally has assessed the extent to which an 
offender needs steady employment to help him avoid reoffending. It now has 
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further tools to evaluate the offender’s aptitude for and interest in developing 
work skills. This should help identify those who would be served best by 
employment programs. 


460 Placing the offender in an appropriate program. The offender case 
management team in the institution uses the intake assessment results to 
refer offenders to appropriate employment opportunities while they are 
incarcerated. However, in the current system there are more offenders with 
identified employment needs than there are openings in related programs. If 
employment training is to operate on the same basis as other reintegration 
programs, then within the institution there must be a fundamental shift in 
attitudes about employment for offenders. The Service must have the 
capacity to train for employment all offenders whose assessments show the 
need. Other offenders who may simply want to work or who can provide 
specific skills to an institutional program can then be accommodated if 
capacity permits. 


461 Program quality. Corcan has enhanced the employment programs in 
the institution by introducing new standards of work, reflecting job 
performance requirements that are consistent with industry standards. It is 
planning to realign all Correctional Service programs of employment and job 
skills training to increase the “job readiness” skills of offenders with identified 
employment needs. For example, it will offer general work skills that will 
make the offender more employable (punctuality, problem solving, teamwork) 
and practical skills (food preparation, carpentry) through on-the-job training. 


462 Evaluating the offender’s progress. Corcan has developed a 
standardized tool for evaluating the extent to which an offender’s 
participation in employment programs has made him more employable. This 
Offender Performance Evaluation is currently being implemented across the 
country. The offender’s progress will be monitored and reported regularly. 


463 Follow-up in the community. As part of the Effective Community 
Corrections initiative, Corcan received $5 million to improve its employment 
services to offenders in the community. It has increased the number of 
employment counselling offices from 12 to 25 across most major urban 
centres. Corcan reported in its Annual Report for 2001-02 that it helped 
1,100 offenders find employment and enrolled 500 offenders in full-time 
education. However, it has no way to identify which regions, institutions, and 
programs have successfully trained offenders and then placed them in jobs in 
the community. 


Vocational training for offenders needs to be restructured 


4.64 Vocational training is one component of employment programs. Even 
though it parallels what Corcan offers to offenders, traditionally it has been 
managed independent of Corcan in the institution. Vocational training 
typically teaches specific skills such as welding or carpentry, and in some 
institutions offenders can obtain external certification of skills they have 
learned. 
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4.65 In April 2002, Correctional Service made Corcan responsible for 
vocational training in order to integrate it better with other components of 
the Service’s Employment and Employability Program. Corcan estimates that 
vocational training programs serving about 550 offenders across all regions 
cost $3 million to $4 million a year. 


466 | Corcan presented a position paper to Correctional Service’s senior 
management on restructuring vocational training. However, no strategy for 
vocational training has been approved and no funds have been allocated. 


467 Recommendation. Correctional Service Canada should fully 
implement employment training of offenders as a core rehabilitation program, 
with particular attention to improving the provision of employment programs 
and vocational training to meet the identified needs of offenders. 


Correctional Service’s response. Action was taken effective | April 2003 to 
require that the results of a specialized employment assessment, which was 
developed in 2002, be addressed in each offender's correctional plan. 
Offenders with identified employment deficits will be referred to appropriate 
employment and vocational programs. 


There are not enough programs for offenders in the community 


4.68 We found that once offenders are released into the community, there 
are not enough programs to follow up on those taken inside the institution. 
Research shows that many intervention programs that deal with factors 
contributing to criminal behaviour are more effective when delivered in the 
community than in an institution. 


469 Our audits in 1996 and 1999 noted that many programs required by 
offenders in the community were not available. For example, in 1999 the 
Service estimated that fewer than 20 percent of released offenders who had 
substance abuse problems were receiving the treatment programs they 
needed. We recommended that the Service ensure an appropriate balance of 
intervention programs between the institution and the community. 


410 The Service has indicated that it has somewhat increased the 
availability of community programs but problems persist. In 2002, using the 
same test it conducted in 1999, Correctional Service estimated that only 
about 1,000 (45 percent) of the 2,200 offenders in the community who 
needed a substance abuse program were receiving this treatment. Waiting 


lists are a concern. 


471 Exhibit 4.7 outlines expenditures for 1998-99 and 2001-02 on 
reintegration programs for offenders, both in the community and in the 
institutions. At first glance, spending on community programs increased from 
about $11 million to about $16 million from 1998-99 to 2001-02. Despite 
this increase, spending on community intervention programs has stayed at 
about 17 percent of the total spending on all intervention programs. 
However, it should be noted that there are more high-risk offenders in 
institutions who require more expensive, high-intensity programs. 
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Exhibit 4.7 Spending on reintegration programs for offenders (1998—99 and 2001-02) 


Expenditures ($000) 2 
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Note: Figures include reintegration program expenditures for both male and female offenders. Psychology 
also includes assessment and a variety of mental health services not directly related to offender 
reintegration programs. 


Source: Correctional Service Canada 


472 Offering follow-up programs for offenders in the community presents 
critical challenges. The duration of programs is long and their admission dates 
are fixed; they are offered only during the day and typically only in larger 
urban centres; offenders may move to another location; and there may not be 
enough participants to warrant offering a program. Some of these challenges 
relate directly to using the same structure and approach for programs in the 
community as for those in institutions, where the Service has more control 
over conditions. 


473 As part of the Effective Community Corrections initiative, the Service 
undertook a $1.6 million, four-year program to deal with these challenges. It 
is developing a Community Maintenance Program with more flexible 
treatment for offenders. Activities funded under the program could address 
some of the known difficulties of delivering programs in the community. 


414. The program is now going into its third year. However, the significant 
lack of community program capacity that existed seven years ago continues 
today. We expected far more progress in this area. 
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4.15 | Recommendation. Correctional Service should set a reasonable target 
to further increase the number of offenders who receive the intervention 
programs they need in the community and should review progress to meet the 
target within a year. 


Correctional Service’s response. Every released offender has a correctional 
plan developed to respond to an assessment of his or her particular risks and 
needs. The Service addresses the offenders' needs through a combination of 
its own programs, community-based programs, specialized services, and other 
interventions. The Service will develop a plan to increase offenders’ 
participation in Service-developed community programming and will monitor 


its progress. 


Supervision of offenders in the community is improving 


476 Supervision in the community is the final step in the process of 
reintegrating offenders—the last point at which the system can directly 
influence or control them. It is when they are under supervision in the 
community that offenders are least separated from the public and therefore 
present the greatest risk to society. When problems occur here, the 
consequences can be severe and are usually highly publicized. 


411 The Service has standards governing the minimum frequency with 
which community parole officers are to meet with offenders. It is important to 
maintain adequate contact with offenders in the community, particularly 
offenders who require a high level of supervision. 


478 In 1999 we tested the Service’s ability to meet its standards for 
frequency of contact with such offenders. We found that it had failed to meet 
its minimum standard in 10 percent to 20 percent of cases, depending on the 
region. We recommended that the Service ensure that its parole officers meet 
the standard for frequency of contact. 


479 Our follow-up found a significant improvement since then in the 
frequency of contact with offenders, the result of a concerted effort by 
management. Correctional Service has measured its progress in a series of 
self-audits, and in 2000-01 it found that in 5 percent to 12 percent of cases 
across all regions, the standard was not met. 


480 In 2002, the Service reviewed the frequency of contact from March to 
July in a representative sample of files on offenders in the community. The 
results indicated a much higher level of adherence to national standards. 
Only about one to four percent of the files, by region, did not meet the 
minimum standard. The field data we reviewed indicated a rate of adherence 
in a range similar to what the Service found in its most recent review. 


481 In its 2002 review the Service also collected information concerning 
the types and levels of “collateral contacts” with offenders in the community 
(collateral contacts are contacts by the community parole officer with the 
offender’s family members, employer, psychologist, program delivery officers, 
and the police, among others). They found that over the review period a large 
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number of collateral contacts had occurred. Among other things, collateral 
contacts add value because they enhance the parole officer’s knowledge of 
the offender and his level of risk. 


Approaches vary in managing offenders who need a high level of supervision 


4.82 In 1994 and in 1999, we noted that the Service was using several 
different approaches to manage offenders who needed a high level of 
supervision in the community. However, it had not evaluated the 
effectiveness of the different approaches. 


4.83 We found in this follow-up audit that approaches to managing 
offenders who need a high level of supervision still vary widely from one 
location to another. In major urban centres, team supervision units handle 
these cases. In other areas, parole officers are assigned equal numbers of these 
offenders. Some area offices assign this group only to more experienced staff. 


484 Correctional Service recently reviewed the approaches it uses to 
manage offenders who require a high level of supervision in the community. It 
has approved national practices for supervising those who may require more 
structure and control. These practices outline, for example, criteria to 
identify the target group, the required frequency of contact, collateral 
contacts, and an appropriate approach to treatment. However, they do not 
define the experience and competency that parole officers need to manage 
these cases. 


4.85 We had also recommended previously that in evaluating its approaches 
to managing offenders who need a lot of supervision, the Service identify and 
implement the approaches that are the most cost-effective in various specific 
circumstances. The Service completed a preliminary evaluation of its 
community supervision practices in October 2002, six months after 
implementing its national guidelines. 


Rate of institutional parole officer turnover appears high 


4.86 During our audit, regional managers and staff frequently mentioned 
the high rate of movement, rotation, and turnover of parole officers. They 
attributed the instability of the workforce to factors such as the frequent 
rotation of staff within and between institutions and community offices, the 
number of job vacancies, and the number of acting appointments. A variety 
of factors affect staff turnover. Operational managers can control some of 
them by, for example, reassigning parole officers to address changes in 
workload. Others are generated by staff themselves, such as departures or 
leave for family-related reasons. 


4.87 Each staff move creates a chain reaction—work must be reallocated, 
temporary acting replacements found, the long process of staffing a 
permanent replacement initiated, and both the temporary acting replacement 
and the future incumbent trained. As we have noted in this chapter, training 
of parole officers and quality control need improvement, issues that heighten 
the impact of high staff turnover on operational stability and case 
management. 
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488 Senior management at Correctional Service headquarters 
acknowledged some concern about the high staff turnover and the use of 
acting appointments. While it had collected few data to assess the extent and 
impact of parole officer turnover, the Service had examined term and acting 
positions in order to reduce their number. With the Service’s assistance, we 
collected some preliminary data on institutional parole officers from its 
human resources data system to assess whether these concerns had validity. 


489 The information we received on all 41 institutions with six or more 
parole officers showed that from 1999 to 2002, parole officers had an average 
annual turnover rate of about 14 percent. Eleven of the institutions had 
annual turnover rates of 20 percent or higher among parole officers. 


490 In June 2002 we also examined data on the number of appointments to 
act in parole officer positions. These data are important because employees 
appointed to act in positions may not have all the knowledge and skills the 
positions require. In the 41 institutions we examined, about 14 percent of all 
parole officers were in acting appointments. Furthermore, in 12 of the 
institutions at least 20 percent of parole officers were acting in their positions. 


4.91 Our review of turnover rates and acting appointments among parole 
officers was limited, as it was based on unaudited data from headquarters. 
However, the preliminary results did reinforce some of the concerns raised 
during our field visits, particularly the high rate of parole officer moves within 
some institutions. High turnover and the use of acting appointments are 
interrelated: lower staff turnover would reduce the need for acting 
replacements. 


4.92 Recommendation. Correctional Service Canada should more 
thoroughly examine the turnover rate among parole officers, assess the 
impact of turnover and how it is being managed, and take the steps needed to 
address its findings. 


Correctional Service’s response. Correctional Service will more closely 
examine the turnover rate of parole officers and take appropriate action 


where necessary. 


Training for parole officers needs improvement 


493 The cornerstone of effective offender reintegration is the knowledge 
and competency of parole officers. It is they who ensure that offenders are 
properly assessed, accurate reports are prepared, and offenders get the 
supervision they need in the community. In 1996 we found that training for 
parole officers was inadequate. We noted that many had not received the 
eight-day orientation training for parole officers until after they had started 
the job. By 1999, the Service indicated that it was now providing parole 
officers 10 days of orientation training, and more of them were participating. 
It was also planning to provide refresher training to experienced parole 
officers. 


4.94 In our current follow-up audit we noted that orientation training for 
parole officers still runs 10 days. This training is essential to provide new 
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parole officers with basic, job-specific knowledge of the reintegration process 
from the intake of offenders to their supervision in the community. Our field 
work found that many new parole officers still do not receive orientation 
training before they start the job. A recent review by the Service showed that 
20 percent of parole officers appointed in the last year did not receive 
orientation training and an additional 15 percent did not receive it in the 
time required. These results clearly indicate the need for substantive 
improvement. 


4.95 Parole officers in the community told us that the 10 days of orientation 
training did not cover all the areas of their jobs adequately. Only the last two 
days of the course dealt with community supervision, and the content was 
considered superficial. 


4.96 As we have noted in this chapter, inter-rater reliability of assessment 
tools used at offender intake is critical to determining the offender’s level of 
security and needs for effective reintegration. Sufficient parole officer training 
and learning are key ingredients of inter-rater reliability. We looked at how 
well current orientation training for parole officers taught and tested these 
competencies. We found that parole officer orientation provided only a 
practical overview of these critical instruments. By comparison, a similar 
offender assessment instrument (the Level of Service Inventory) used in 
numerous jurisdictions worldwide (United States, Britain, and Australia) 
requires more extensive training and officer testing to ensure that it is applied 
as intended. 


4.97 Correctional Service’s national headquarters has not given the regions 
detailed guidelines for structured on-the-job training, and we found that 

training on the job is uneven. Only some offices provide structured training; 
most training on the job is provided by the more experienced parole officers. 


4.98 ‘To be effective on the job, parole officers need continued training to 
enhance their skills. This can include training in risk assessment, supervision 
of sex offenders, crisis management, defusing of hostile situations, and 
negotiation. Correctional Service has not identified those who should receive 
this enhanced training and when. 


4.99 Some parole officers have not received the enhanced training they 

need. Most told us that it is left up to parole officers to seek out enhanced 
training on their own initiative in the annual review of their performance. 
Even then, finding funds and time for training can be difficult. 


4.100 ‘To address the known concerns about the training of parole officers, 
senior management recently approved five additional days of development 
each year, starting in 2003-04. The training will be developed nationally to 
enhance parole officers’ understanding and application of effective 
correctional practices. The Service says that three days of the training will be 
in the classroom and the two final days will be a combination of on-line 
training, conference attendance, and external courses. 


4.101 The Service has also increased its funding for staff training, and it will 
provide added specialized training on-line to enhance the present orientation 
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training for parole officers. In addition, it provides supervisors of parole 
officers with professional development to enhance the quality of community 
supervision and will provide safety training for parole officers in the 
community. 


4.102 Recommendation. Building on its recent efforts, Correctional Service 
Canada should ensure that all parole officers working at different points in 
the reintegration process (at offender intake, in the institution, and in the 
community) receive the training they need to perform their specific duties. 


Correctional Service’s response. The Service has already recognized this 
need and, to that end, it has introduced a national training standard of five 
days annual training, which will provide parole officers with additional 
specialized training on the clinical aspects of their duties. 


Conclusion 


4.103 In 1999 we concluded that Correctional Service was moving in the 
right direction with initiatives to make its practices and results more 
consistent across all of its regions. 


4.104 Today, the Service has made good progress in several areas that we 
have raised previously: 
¢ It takes less time to acquire critical documents. 


* It has more time to complete the case management process for offenders 
in institutions. 


¢ The quality of reports to the National Parole Board continues to 
improve. 


¢ It has made a significant improvement in meeting its standards for 
frequency of contact with offenders in the community. 


¢ It has developed and evaluated national guidance on supervising 
offenders who need a high level of supervision in the community. 
4.105 However, the Service needs to improve further in other areas where we 


made recommendations in 1999. It still needs to 


* provide enough programs for offenders in the community when they 
need them; 


* implement its strategy for employment and vocational programs; and 
* ensure that senior correctional officers fulfil their duties in case 


management. 


4.106 During this follow-up we also identified some related areas that need 
attention where we did not make recommendations in 1999. Correctional 


Service needs to 


* thoroughly test the reliability of the tools it uses to assess offenders at 


intake; 
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* improve the training of parole officers to meet the requirements of the 
job; 
* assess and address the apparently high turnover of parole officers; and 


* complete offenders’ correctional plans to meet existing standards. 


4.107 A future direction for reintegration of offenders. Correctional 
Service has modified the structure of its reintegration process several times 
over the past decade. In September 2002 it approved the latest step in the 
evolution, called “operating regimes.” This is a new approach to help specific 
groups of offenders achieve their correctional goals with the aid of a 
specialized team of correctional staff. A key part of this new model is to 
increase offenders’ responsibility for their behaviour and for their active 
involvement in achieving the objectives of their correctional plans. 


4.108 Several operating regimes are planned: one for highly disruptive 
offenders in maximum security institutions; a transition regime for offenders 
preparing for release to the community; and a number of separate operating 
regimes in the community based on offenders’ needs and potential for 
reintegration. However, operating regimes will succeed only to the extent that 
the Service builds on the improvements it has already made in its 
reintegration practices and addresses those areas we have identified that still 
need improvement. 
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About the Follow-up 


Objective 


The objective of this follow-up audit was to assess the extent to which Correctional Service Canada has acted on 
recommendations made in Chapter | of our 1999 Report. Specifically, we assessed the extent of improvement in the 
following areas of offender reintegration: 

* official documentation received at intake, 

* reliability of risk assessment tools, 

* timeliness of case management, 

* quality of reports to the National Parole Board, 

* availability of programs for offenders in the community, 

* implementation of the employment strategy, and 


* supervision of offenders in the community 
Audit scope 


Our examination focussed on the critical components of the reintegration process for male offenders that contribute 
to effectively reintegrating offenders into the community as law-abiding citizens. We also examined the extent to 
which parole officers working with offenders throughout the reintegration process have the necessary knowledge, 
skills, and experience to perform the duties of their jobs. The audit did not include issues specific to male Aboriginal 
offenders. 


Audit criteria 


We expected the following: 


¢ Risk assessment instruments used to identify an offender’s criminogenic factors and level of security throughout 
the sentence are valid, reliable, and properly applied. 


* Accurate, timely, and complete information on the offender both from external sources and from within 
Correctional Service are available to complete the offender’s correctional plan. 


* Good-quality, timely intervention and employment programs are provided by qualified individuals to offenders 
while they are incarcerated and while under supervision in the community. 


* Information on offenders’ progress against their correctional plans is collected on an ongoing, timely basis and is 
accurate and complete. 


* The Service regularly analyzes the reasons why offenders remain incarcerated past their first date of parole 
eligibility. 

* Complete, timely, and accurate reports are provided to the National Parole Board to grant conditional release 
to offenders at their first date of parole eligibility and at subsequent reviews. 


* Parole officers maintain contact with and supervise offenders in the community at a level that is appropriate 
and consistent. 

* Parole officers working at different points in the reintegration process (at intake, in the institution, and in the 
community) possess the necessary knowledge, skills, and experience to perform the tasks and duties of their 
jobs. 

* Correctional Service has the accurate and timely performance information it needs on all operational aspects of 
the reintegration process, including human resource indicators, to make informed management decisions. 
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Audit team 


Assistant Auditor General: Hugh McRoberts 
Principal: Ronald Wolchuk 


William Johnson 
Dawn-Alee Fowler 
Mary Lamberti 
Dan Thompson 
Stacey Wowchuk 


For information, please contact Communications at (613) 995-3708 or 1-888-761-5953 (toll-free) 
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Appendix Offender Population, Admission, and Release Trends (1997-98 to 2001-02) 


Offender Population Offender location 1997-98 1 1999-00 2000-01 2001-02 
Institution ! Us} M2) 12,776 12,474 12,419 12,306 
(61%) (60%) (59%) (59%) (60%) 
Community * 6,239 8,656 Si SZ 8,579 3,275 
(39%) (40%) (41%) (41%) (40%) 
Total ZFS OO 21,432 21,266 20,998 20,582 


Note: ! Institutional figures do not include escaped offenders (154 in 2001-02). 


@ Community figures do not include offenders deported upon release and provincial offenders under federal supervision. 
Included are offenders unlawfully at large (646 for 2001-02) and offenders on parole who have been temporarily detained 


(850 for 2001-02). 


Offender Admission Type of Admission | 1997-98 1998-99 1999-00 2000-01 2001-02 
Warrant of committal 4,224 4,418 | 4,128 4,057 | 3,926 
Revocation* without offence DADS I fekay2 1 Gal 1 ee! 1,704 
Revocation* with offence 1,004 : 1,105 | 1,108 ce 13) ies 962 ; 
Revocation* with outstanding _ 7 5) 287 | SHAG 
charge 
Total admissions V5 Adil 7.395 7,192 LASS 6,962 
Note: *Conditional release suspended and offender returned to federal prison. 

Offender Releases Type of Release 1997-98 1998-99 1999-00 2000-01 2001-02 
Day parole 7 DATA | 2673 | 2659 2,305 2,084 
Full parole! 540 396 290 PANES) 224 
Statutory release SM TIS 4,340 4,429 4,568 4,690 
Warrant expiry 398 333 266 214 196 
Total 8,185 7,742 7,644 7,302 7,194 
Note: ! These figures do not include offenders who change from day parole to full parole. 

They are shown below. 
: 1997-98 1998-99 1999-00 | 2000-01 | 2001-02 
Day parole to full parole | 1,034 | IL ASC 1,147 | 932 | 837 
Source: Correctional Service Canada 
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Chapter 


Post-Secondary Recruitment Program of 
the Federal Public Service 


All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 
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Results of the program changes 


The program’s contribution to new hiring has stayed about the same 

Many managers still do not use the program 

Short-term hiring practices remain a problem 

The program works well to recruit for development programs 
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Post-Secondary Recruitment Program 


of the Federal Public Service 


Main Points 


5.1 The Public Service Commission has redesigned the Post-Secondary 
Recruitment program to better meet the needs of both students and 
government departments and has improved the way it markets the program 
to universities. However, we found that departments use the program 
relatively infrequently to staff positions in their regional offices. 


5.2 The Public Service Commission has substantially implemented the 
changes to the program that we recommended in our December 2000 Report. 
Over the last three years, departments have filled increasing numbers of jobs 
through the Post-Secondary Recruitment program. However, the increase has 
been minimal in proportion to the increase in total numbers of people hired 
into the public service during the same period. 


5.3. In our December 2000 Report, Chapter 21, we recommended that the 
government develop a results-oriented recruitment strategy that would 
identify post-secondary recruitment targets to address workforce renewal 
challenges for the years ahead. In our 2002 follow-up work discussed here, we 
found that some departments and the government as a whole have made 
limited progress in human resources planning and in establishing recruitment 
targets. We found that some departments and the government as a whole 
have not analyzed their recruitment and renewal needs. Nor does the 
government have a complete picture of the educated and skilled people who 
are entering the public service through its various recruitment routes. 


5.4 Many managers continue to adopt the expediency of short-term hiring, 
despite the opportunity to recruit through a fast and flexible program. 
Departments need a balance between hiring to fill immediate vacancies and 
hiring strategically, with a view to ensuring that the government will have 
qualified people to deliver its programs and services in the future. 


The Public Service Commission, the Treasury Board Secretariat, and the 
departments we audited have responded. The Public Service Commission, 
the Treasury Board Secretariat, and the departments we audited agree with 
the recommendations of our follow-up. Responses throughout the chapter 
indicate measures being taken or planned to address the recommendations. 
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Introduction 


A federal public service facing renewal needs 


§.5 The federal public service will not have the capacity to deliver services 
if it does not attract people with high-level skills and education. An aging 
workforce, changes in the nature of work, and competition for resources are 
challenges that need to be addressed. 


5.6 The aging of Canada’s population is reflected in the make-up of the 
federal public service. According to the Treasury Board Secretariat’s 
forecasting models, the proportion of employees aged 55 and over is expected 
to increase over the next 10 years. Particularly in the upper and middle ranks, 
many are approaching the time when they will be eligible to retire. 


5.7 Atthe same time, the profile of jobs in the public service is undergoing 
a transformation to more scientific, technical, and professional positions that 
require higher levels of education and skills. Replenishing these positions is 
critical to maintaining a well-functioning public service. Departments are 
concerned about shortages in the technical and scientific occupations and 
the management and leadership categories. 


5.8 Between 1999 and 2002, speeches from the Throne, the Clerk of the 
Privy Council’s annual reports to the Prime Minister on the public service of 
Canada, and the Prime Minister’s responses to the Clerk have reiterated the 
importance of attracting and developing the talent needed to serve 
Canadians in the 21st century. Following the hearings on our December 2000 
Report, Chapter 21, and our December 2001 Report, chapters 2 and 3, the 
Standing Committee on Public Accounts concluded that the government's 
capacity to attract talented and skilled people needed significant 
improvement. 


The Post-Secondary Recruitment program 


59 The Post-Secondary Recruitment program is managed by the Public 
Service Commission. It is the federal government’s main structured program 
for recruiting graduates into a wide variety of entry-level positions in the 
Administrative and Foreign Service category and the Scientific and 
Professional category. It is also an excellent source of candidates to renew and 
rejuvenate the federal workforce and help address the urgent challenge of 
recruiting for the years ahead. 


5.10 At this time, the government has introduced new legislation to 
modernize the public service of Canada that proposes delegating authority for 
staffing to departments. We do not yet know the implications for centrally run 
programs. Exhibit 5.1 shows how graduates may currently enter the federal 
public service through the Post-Secondary Recruitment program and other 


avenues. 


5.11 The Post-Secondary Recruitment program has a number of positive 
features. Its selection processes are based on an assessment of whether an 
applicant’s education fits the job. The program has consistently met or 
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exceeded the federal public service’s employment equity targets except for 
people with disabilities. It is open to applicants from all over Canada, and it 
offers jobs that are mostly indeterminate (continuing employment without a 
specified end date) and linked to long-term public service careers. 


Exhibit 5.1 How university graduates enter the federal public service 


General recruitment 


Individual competitions for vacant positions in departments for both short- and long- 
term employment in all occupational groups. 


Competitions managed by the Public Service Commission or by departments with 
special delegated authority to recruit for highly specialized jobs, such as nurses and 
doctors at Health Canada and Veteran Affairs Canada and economists at the 
Department of Finance. 


Post-Secondary Recruitment program 


Recruitment to specific entry-level positions in departments—managed by the Public 
Service Commission for the Administrative and Foreign Service and the Scientific and 
Professional occupational groups; mainly for long-term career appointments. 


Recruitment to entry-level positions for development programs—managed by the 
departments, for example, programs at Statistics Canada for economists/sociologists 
and mathematical statisticians; Public Works and Government Services Canada for 
supply officer trainees, and Foreign Affairs and International Trade for the Foreign 
Service and for management and consular affairs officers. 


Recruitment to entry-level positions for corporate development programs—managed by 
the Public Service Commission: the Management Trainee program, the Accelerated 
Economist Training program, the Policy Research Development program, and the 
Human Resources Development program. 


Recruitment to entry-level positions for the Financial Officer Recruitment and 
Development program and the Internal Auditor Recruitment and Development 
program—managed by the Treasury Board Secretariat. 


Student bridging 


Students who have worked for the federal government under an official student 
program and who are hired through the bridging program after graduation. 


Managed by departments for all occupational groups; indeterminate or term status 
employment. 


Focus of the follow-up 


§.12 Our December 2000 Report (Chapter 21) discussed our audit of how 
the Public Service Commission was managing the Post-Secondary 
Recruitment program and to what extent federal departments were using the 
program. This follow-up audit examined the government's progress in 
addressing our recommendations from that chapter. As part of the follow-up, 
we looked at changes to the program and at their impact. We looked at 
whether more departments are using the program as part of their strategy for 
human resources renewal and recruiting more new graduates into their 
workforce, particularly in their regional offices. We also examined whether 
the Public Service Commission has improved the way it markets the program 
to universities. Finally, we examined whether the Commission has improved 
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the way it reports the program’s results to Parliament. We did not audit the 
Commission’s quantitative data on the Post-Secondary Recruitment program; 
however, we assessed the Commission’s data compilation methods. 


5.13 Our 2000 audit included three departments as examples of users of the 
Post-Secondary Recruitment program: the Department of Foreign Affairs and 
International Trade, Fisheries and Oceans Canada, and Statistics Canada. For 
the follow-up, we added three departments: Public Works and Government 
Services Canada (PWGSC), Human Resources Development Canada 
(HRDC), and Veterans Affairs Canada. We chose PWGSC and HRDC 
because they are very large departments with a sizeable presence in all regions 
of Canada. We chose Veterans Affairs because its headquarters are outside 
the National Capital Region and we were interested in the regional 
recruitment practices of departments in this situation. Our follow-up scope 
and criteria are set out in About the Follow-up at the end of the chapter. 


Observations and Recommendations 


Changes in the design of the program represent progress 


5.14. Our December 2000 Report made a number of recommendations to 
improve the Post-Secondary Recruitment program. We recommended that 
the Public Service Commission create inventories of pre-screened, qualified 
candidates to speed up the staffing process and encourage managers to hire 
graduates from the inventory to indeterminate positions. We said that it 
needed to improve the scheduling and timeliness of the program’s 
recruitment campaigns. And we recommended that the Commission and 
departments systematically assess recruitment activities under the program 
and report to Parliament on the program’s use. 


§.15 Following the hearing on our Report, the Standing Committee on 
Public Accounts also made a number of recommendations to the 
Commission. They included enhancing opportunities through the program 
for recruitment in all regions across Canada, developing a database of 
pre-screened candidates, and reporting to Parliament on improvements made 
and the results achieved. 


5.16 The Public Service Commission redesigned the program as a result. 
Earlier in 2000, the Commission and the Treasury Board Secretariat had 
published a Recruitment Action Plan, which called for the government to 
improve its human resources planning and to set recruitment targets. It 
recommended that departments use the Post-Secondary Recruitment 
program to fill more positions, that the Commission and departments 
co-ordinate their marketing of the program on university campuses, and that 
the marketing promote the government as a workplace of choice. 


5.17 The differences between the former program and the redesigned 
program are shown in Exhibit 5.2. The program’s application process has been 
entirely electronic since the fall 2001 recruiting campaign, allowing 
candidates to apply directly through the Internet. 
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Exhibit 5.2 Changes in the Post-Secondary Recruitment program 


Former program 


Redesigned program 


Up to 1999, the Post-Secondary Recruitment program was an 
annual campaign receiving applications only during one month 
in the fall. In 2000 and 2001 there were winter and fall 
campaigns. 


The program runs throughout the year. 


Departments were restricted to advertising during the one- 
month campaign period. 


At any time of the year, departments can advertise positions 
requiring a recent university degree. 


Periods for applying were restricted. 


Candidates can apply to the inventory or to departments’ current 
posters at any time. 


Applications with missing data risked being rejected. 


Applicants can provide missing data at a later date. 


Departments had to rely on their own posters and could pick 
only from graduates who applied during the campaign periods. 


Departments can select suitably qualified candidates from the 
inventory whenever they are ready to staff a position. 


Prior to May 2001, departments could post jobs through the 
Post-Secondary Recruitment program only if there were five or 
more positions. Subsequently, to encourage use of the program, 
departments could advertise single positions. 


Using the Post-Secondary Recruitment program’s inventory 
allows one-stop shopping for candidates seeking similar 
positions in a number of departments. 


Candidates not selected during a particular campaign would 
have to re-apply during the next campaign if they were 
interested in, and met the requirements of, the positions 
advertised. 


Candidates can choose to keep their applications active by 
reconfirming their application every 60 days. 


Screening tests were administered only in association with the 
annual or biannual campaigns. 


Candidates can apply to specific posters and the inventory at 
any time. 


Applicants needed to take whatever tests were required for the 
job for which they were applying. 


Two tests, the Graduate Recruitment Test (GRT) and the Written 
Communication Proficiency Test (WCPT), are currently used to 
screen in the stronger candidates and are administered at 
regular intervals. 


Departments made their own selection from applicants to their 
own posters. 


Candidates applying to the inventory can write the GRT and 
WCPT to complete their applications and be readily available for 
consideration by departments. Taking these tests is optional but 
is recommended by the Public Service Commission. 
Departments may choose to require further testing. 


Departments had a heavy workload of candidate screening. 


A pre-screened inventory allows a department to request the 
Public Service Commission to refer the top 10 candidates 
meeting specific requirements. This reduces departments’ 
workload considerably. Departments must still carry out their 
own screening of applicants responding to their own posters. 


Applications were received during the campaign month, 
followed by mass testing, followed by referral to departments. 
Paper referrals were sent to departments. 


A faster, electronic referral process takes under 48 hours for 
career choices requiring no further testing. 


Tests were evaluated manually by correctors. 


Tests are corrected electronically—95 percent of tests are 
corrected in two weeks. 


Traditional testing methods were used. Applicants sat for exams 
at a designated centre at an appointed time. 


On-line testing of candidates is planned (successfully piloted in 
Edmonton in fall 2001). 
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5.18 Two fundamental changes are the transformation from a twice-yearly 
campaign to a continuous campaign and the creation of the Post-Secondary 
Recruitment program inventory. 


The Public Service Commission has improved the way it markets the program on 
campuses 


5.19 In our 2000 Report, we observed that the Public Service Commission 
needed to be a better recruiter: it was not promoting a career in the public 
service sufficiently and aggressively. Nor was it working with departments to 
co-ordinate their visits to university campuses. In response to our 
observations, the Commission said it had allocated resources to enhance its 
promotion and marketing efforts. In our follow-up we expected to find that 
the Post-Secondary Recruitment program was more visible on campuses and 
that students had a better perception of the career opportunities the public 
service has to offer. 


§.20 The majority of university career advisors and recruitment 
co-ordinators we interviewed knew about the redesign of the program and 
liked the changes. We found that their perception of the public service as an 
employer had improved. In part, this was a response to changes in the 
economy, but it was also because the Commission had improved its marketing 
methods and materials. It continues to publicize the program on campuses by 
attending career fairs and communicating with program directors and deans. 
University career advisors and recruitment co-ordinators noted an 
improvement in the appearance of the Commission’s kiosk, which features 
electronic access to the jobs.gc.ca Web site. The Web site itself is user- 
friendly. The kiosks are staffed by the Commission’s local human resources 
personnel who are responsible for student recruitment and have the 
knowledge to respond to students’ questions. 


5.21. In 2001, the Commission added a Student Ambassador Initiative to its 
marketing strategy. Student ambassadors are full-time students who are paid a 
small salary to work 8 to 10 hours weekly throughout the academic year, 
promoting public service employment to their fellow students. University 
career advisors and recruitment co-ordinators on campuses where a student 
ambassador was present reported that the initiative was succeeding in 
educating students about careers in the federal government. 


5.22. In our 2000 audit, we found that students were confused about who 
was doing the hiring: the Public Service Commission, a department, or a 
federal agency. The Public Service Commission has tried to solve this problem 
by linking the program’s Web site to other federal government employment 
sites. The Commission also tries to co-ordinate its campus activities with 
those of departments. Although the students’ understanding has improved, it 
is still not clear to them who is recruiting. A study by the Treasury Board 
Secretariat of the marketing and promotional practices that support 
recruitment recommended that the Government of Canada develop a more 
consistent corporate image, but no further action has been taken yet. 
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The program’s contribution to new hiring has stayed about the same 


5.23 According to the Public Service Commission’s data, the number of 
appointments made through the Post-Secondary Recruitment program over 
the last three years shows a significant upward trend. However, the total 
number of indeterminate employees hired into the federal public service has 
also increased over the same period, and the proportional contribution of 
Post-Secondary Recruitment program appointments to new indeterminate 
hiring has remained roughly the same, despite the program’s redesign. 


5.24 Exhibit 5.3 shows that the number of appointments made through the 
program almost doubled between fiscal years 1999-2000 and 2001-02, from 
560 to 1,020; the number of appointments outside the National Capital 
Region also more than doubled, from 125 to 263. 


Exhibit 5.3 Post-Secondary Recruitment program appointments in the National Capital Region and 
other regions 


Program appointments 


1,200 z= 7 ‘ : ; ms HB Other regions 
[J National Capital Region 
[1 Total 


1,000 | 


1999-2000 2000-01 2001-02 


Data include all departmental and corporate programs that recruit through the 
Post-Secondary Recruitment program 


Source: Public Service Commission: Post-Secondary Recruitment program data (not audited) 


5.25 Over the same period, the total number of federal public service 
employees grew by some 18,000, an increase of about 12 percent according to 
the Treasury Board Secretariat’s public service employment statistics. The 
number of new indeterminate employees hired rose from 2,874 in 1999-2000 
to 3,856 in 2000-01, and to 5,046 in 2001-02. Exhibit 5.4 shows that 
appointments through the Post-Secondary Recruitment program remained 
steady at about 20 percent of total indeterminate hiring over the past three 
years. 


5.26 Appointments through the Post-Secondary Recruitment program in 
regions outside the National Capital Region also rose in relation to all 
indeterminate hiring (Exhibit 5.4), from 4 percent in 1999-2000 to a high of 
6 percent in 2000-01; and then it dropped to 5 percent in 2001-02. 


5.27 In our 2000 Report, we said we were concerned about the low rate of 
use of the program in the regions, given that 60 percent of the federal public 
service works outside the National Capital Region. We recommended that 
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the Public Service Commission use the program to offer opportunities across 
the country. Our follow-up found that the number of people hired through 
the program in the regions remains relatively low. 


Exhibit 5.4 Appointments through the Post-Secondary Recruitment program as a percentage of new 
indeterminate hiring 


Percent 
19.5 20 
20 19 


Program appointments as percent 
of total indeterminate hiring 


Ea Program appointments outside 
the National Capital Region as percent 
of total indeterminate hiring 


15 


10 


1999-2000 2000-2001 2001-2002 


Source: Public Service Commission: Post-Secondary Recruitment program data and annual report 
hiring statistics (not audited) 


Many managers still do not use the program 


5.28 The Public Service Commission’s appointment statistics and the 
progress reports of its and the Treasury Board Secretariat's Recruitment 
Action Plan show that the number of departments using the program 
increased from 26 in 1999-2000 to 32 in 2000-01, and to 38 in 2001-02. 
These 38 departments together make up about 97 percent of the total public 
service population. Thus, what is needed is not so much for more 
departments to use the program but for those that already do to use it more, 
especially in the regions. 


§.29 During our audit of the program in 2000 and our audit of general 
recruitment one year later (Auditor General’s December 2001 Report, 
Chapter 3), managers told us they needed better tools to identify candidates 
for recruitment, including inventories of pre-qualified individuals from which 
they can quickly identify those who meet their needs. Managers also 
complained then about the long lead time needed to post positions under the 
former program, because they could not always predict job vacancies that far 
ahead. In our opinion, the changes to the program have addressed these 
problems. 


5.30 The human resources personnel we interviewed both at departmental 
headquarters and in the regions welcomed the changes to the program but 
reported that many of their line managers gave them a number of reasons for 
not using the program and the inventory. A primary reason was that the 
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inventory is new, and some managers are not yet fully aware of it and how it 
can reduce their recruitment workload and the time it takes to recruit. 


5.31 Further, some managers prefer to use the student bridging program 
because they have already invested in training the students and they know 
their work. In the bridging program, students who have worked for the federal 
government to gain job experience during their studies can be hired after 
graduation either with or without a job competition, depending on the initial 
selection process. 


5.32. However, the Public Service Commission is concerned that managers 
in many departments and agencies are not using the Post-Secondary 
Recruitment program enough to meet their renewal needs. A series of letters 
from the Commission to heads of personnel in departments and a letter dated 
July 2002 from the President of the Commission to deputy ministers and 
agency heads encouraged departments and agencies to increase the number 
of jobs available to graduates through the program, particularly in the regions. 
The Commission also uses presentations to the human resources community 
to call for more appointments through the program. 


5.33. Three of the six departments covered in our follow-up audit have a 
sizeable presence in the regions outside their headquarters in the National 
Capital Region: Fisheries and Oceans Canada, Public Works and 
Government Services Canada, and Human Resources Development Canada. 
We found that their regional offices make only limited use of the program, 
although Fisheries and Oceans Canada’s central Post-Secondary Recruitment 
program campaign in 2001 (for about 15 positions) took the Department’s 
regional needs into account. We also noted a line manager in one of the 
regional offices of Public Works and Government Services Canada who had 
undertaken a special recruitment initiative through the program. 


5.34 Human resources personnel in these departments’ regional offices gave 
us various reasons why most of their line managers did not use the program. 
These included the fact that they had a stable workforce or that they did not 
need university graduates for the jobs that became vacant. 


5.35 Veterans Affairs Canada, which has its headquarters in Prince Edward 
Island, said it has made little use of the program, in part because it has its own 
delegated authority to hire doctors and nurses for its hospital facilities. 
However, it does participate in the Post-Secondary Recruitment program’s 
corporate development programs (as described in Exhibit 5.1) from time to 
time to recruit people in other fields. 


5.36 The Commission is also worried about underuse of the Post-Secondary 
Recruitment program’s inventory by regions outside the National Capital 
Region. As an additional initiative to promote the program’s inventory to 
departments, the Commission recently instructed its Post-Secondary 
Recruitment program co-ordinators throughout the country to encourage 
departments to check the inventory for candidates to meet their general 
recruitment needs. In December 2002, the inventory listed 9,000 graduates 
or students about to graduate, 6,000 of whom had passed the Graduate 
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Recruitment Test and the Written Communication Proficiency Test, the two 
basic tests used to screen in the stronger candidates. Yet the total number of 
appointments made through the program in the previous year was around 
1,000, and most were in the National Capital Region. University career 
advisors and recruitment co-ordinators caution that the inventory will lose 
credibility if students hear how few appointments are made from it. 


5.37 We expected that the changes to the Post-Secondary Recruitment 
program, added to the program’s impartial selection process, its national 
character, and its good record in meeting employment equity targets, would 
have led to more extensive use of the program, especially in the regions. We 
share the Public Service Commission's concerns about departments not using 
the redesigned program as much as anticipated. 


Short-term hiring practices remain a problem 


5.38 In the follow-up we were told that managers use short-term hiring 
because their departments do not have human resources plans and do not 
budget for more than a year at a time. Managers are also under pressure to 
complete their work on time, so they prefer to hire people who already have 
experience. The hiring managers who took part in the survey we conducted 
during our audit of general recruitment (December 2001 Report, Chapter 3) 
made similar comments. 


5.39 In that audit chapter we referred to a culture of short-term hiring and 
said that recruitment practices needed to change. The recruitment system 
had to be faster and more flexible. Managers needed to change their mindset 
on recruitment to ensure a better balance between the immediate need to fill 
vacancies and the strategic needs of their departments. Though the Public 
Service Commission has redesigned the Post-Secondary Recruitment program 
since then to make it more nimble, our follow-up work indicated that many 
managers have not altered their outlook on recruitment. 


§.40 A joint study on term employment published by the Public Service 
Alliance of Canada and the Treasury Board Secretariat in December 2002 
showed the problems caused by the use of term hiring for other than 
legitimate short-term purposes. Examples of legitimate uses are filling 
temporary vacancies while indeterminate employees are on leave or on 
assignment, staffing short-term projects, or assisting with heavier than usual 
workloads. The study confirmed what we had found in 2001: some managers 
use term hiring because they find it more convenient than following the 
current rules for indeterminate hiring. The Treasury Board’s new policy on 
term appointments shows the seriousness of the problem and the need for a 
drastic change in departments’ approach to recruitment. 


The program works well to recruit for development programs 


5.41. Some departments are doing a good job of linking the program to their 
own development programs. Three of the departments in our follow-up— 
Statistics Canada, Public Works and Government Services Canada, and 
Foreign Affairs and International Trade—have used the program to recruit 
for their formal development programs. For the most part, these departments 
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have a good knowledge of their workforce and have looked ahead to their 
future business needs. 


5.42 Statistics Canada establishes its human resources requirements 
through demographic projections and renewal and business planning. As we 
reported in December 2000, this department is highly regarded for its 
recruitment practices and continues to maintain its excellent reputation for 
planning and recruitment. PWGSC also uses demographic analysis to forecast 
its needs, although its human resources managers would like line managers to 
improve their planning of workforce renewal. PWGSC has put measures in 
place to help line managers use the demographic analysis for planning. At 
PWGSC we also found an example of a regional office using the Post- 
Secondary Recruitment program to fill the occupational gaps that its own 
regional human resources plan had identified. Foreign Affairs and 
International Trade relied mainly on its established procedures and traditions. 


5.43 These three departments recognize that they must invest in recruiting 
and developing graduates to continue providing their core services. They 
have senior management support for strategic recruitment through the Post- 
Secondary Recruitment program. 


5.44 At HRDC, one branch recently implemented a development program 
for its core staff and uses the Post-Secondary Recruitment program. In 
contrast, managers in departments with no longer-term renewal strategy hire 
mainly to fill each vacancy as it occurs. 


5.45 The government also uses the Post-Secondary Recruitment program to 
recruit university graduates in specific disciplines for five special development 
programs to meet government-wide needs. These corporate programs set 
targets annually for the number of trainees they aim to hire. We reported in 
2000 that some of the corporate development programs had not met their 
annual recruitment targets. Since then, the number of graduates recruited 
through the corporate programs has increased, and all programs have reached 
or almost reached their targets. The Treasury Board Secretariat’s recent 
Review of Corporate Student Employment and Entry Level Recruitment and 
Training Programs concludes that corporate development programs contribute 
to the renewal of the public service. 


Departments and the government as a whole have not determined their recruitment 
needs 


5.46 Among the six departments involved in our follow-up, the extent of 
their demographic analysis and human resources planning varied 
considerably. Statistics Canada is recognized for its use of sophisticated 
demographic analyses and human resources planning models. It sets 
recruitment targets for its principal occupational groups. Some departments 
had a more accurate knowledge of their headquarters’ and regional offices’ 
recruitment needs and activities than others. For example, HRDC had done 
some recent demographic analysis of its workforce, but it had not updated its 
1998 recruitment and retention strategy and had little knowledge of 
recruitment activities in its regional offices. One region had developed its 
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own strategic human resources plan in 2000 with an update planned in 2003. 
Fisheries and Oceans Canada has completed a demographic analysis of each 
occupational group and has identified where there are likely to be shortages. 
It recognized that it needed to improve its human resources planning 
capability and is working with the Public Service Commission to develop this 
capacity. 


5.47 In both 2000 and 2001 we reported that capacity for human resources 
planning and demographic forecasting varied significantly from one 
department to the next. We said the government needed to develop a 
strategic direction and specific goals for recruitment to meet the renewal 
challenge. Without recruitment strategies, departments found it difficult to 
identify whom they needed to recruit. We were concerned that the 
government had made recruitment a priority without setting targets for 
recruiting to critical occupational groups. 


5.48 The Recruitment Action Plan published in July 2000 by the Public 
Service Commission and the Treasury Board Secretariat called for the 
government to clearly identify public service recruitment needs for the next 
three to five years. The plan listed recruitment forecasts among its short-term 
deliverables. In the two years since, the government has set no 
comprehensive targets, even though several occupational communities are at 
tisk because people with the right qualifications are in short supply. People 
with these special qualifications are often attracted to work for the private 
sector rather than the public sector. The Recruitment Action Plan follow-up 
reports of September 2001 and November 2002 note that certain 
departments have made some progress and use good practices in their 
demographic analysis, human resources planning, and recruitment strategies. 
However, most of them do not report any recruitment targets. The Public 
Service Commission recently began training departments to improve their 
human resources forecasting and planning capacities. 


5.49 The Treasury Board Secretariat’s Review of Corporate Student 
Employment and Entry Level Recruitment and Training Programs found that any 
government renewal strategy would need to include an assessment of 
government-wide demand for entry-level recruitment and training programs. 


The government has no global picture of new graduates hired 


5.50 The Post-Secondary Recruitment program is just one of several ways 
for graduates to begin a public service career. No government-wide tally exists 
to show how many graduates are entering the public service through each of 
the different recruiting avenues, such as student bridging and general 
recruitment, or what types of jobs these graduates accept when they are hired 
and under what status of employment. 


5.51 The Treasury Board Secretariat’s Review of Corporate Student 
Employment and Entry Level Recruitment and Training Programs pointed out 
that there is no inventory of departmental entry-level recruitment and 
training programs across departments—no record of who is doing what. The 
government needs to know how many graduates are entering the public 
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service and with what knowledge and skills. That knowledge and an 
understanding of where gaps in its workforce are likely to occur will help the 
government target development programs at graduates already in the public 
service. It will also help recruitment programs such as the Post-Secondary 
Recruitment program to advertise positions specifically and clearly and to 
make more timely job offers. Once the government has accurate information 
about the career opportunities it has available to graduates, it can market and 
promote its corporate image as a workplace of choice. 


5.52 Recommendation. The government and departments should adopt a 
strategic approach to renewal based on demographic analysis, projections, 
and knowledge of current and anticipated business needs. They should 
develop specific targets for recruitment, particularly to the occupational 
groups that anticipate critical shortages. 


Human Resources Development Canada’s response. HRDC is currently 
undergoing an extensive renewal initiative to modernize service to Canadians 
and to undertake a complete policy and program review, as well as a corporate 
services review. A comprehensive human resources transition strategy will be 
developed to support the renewal agenda. This strategy will take into account 
the realignment of existing business lines and current resources as well as any 
future recruitment needs. 


Public Works and Government Services Canada’s response. PWGSC 
concurs with the recommendation. The Department is aware of the 
importance of building a strong foundation for the future and has undertaken 
a number of strategic human resources initiatives to support this objective, 
including a new integrated business planning cycle in 2001-02; the 
implementation of a comprehensive human resources management 
framework; and training in human resources planning for managers. As well, 
each branch head is provided semi-annually with demographic analyses and 
projections, including key group analyses, and is responsible for developing 
recruitment plans. 


Managers are encouraged to staff positions on an indeterminate basis and to 
use apprenticeship and professional training programs for new recruits, with 
the result that PWGSC has a very low proportion of term employees. 


Given that 70 percent of PWGSC staff are located in the national Capital 
area, and given the nature of regional work and the relatively low number of 
entry level professional positions in the regions, the opportunities for the 
regions to recruit through the Post-Secondary Recruitment program are 
limited. 


Statistics Canada noted that the chapter recognized its excellent reputation 
for demographic analysis, projections, and knowledge of its future needs, and 
therefore offered no further response. The Department of Foreign Affairs and 
International Trade, Veterans Affairs Canada, and Fisheries and Oceans 
Canada acknowledged the chapter, with no further comment. 
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5.53 In our last reports on human resources management issues, we 
criticized the quality of the information provided to Parliament. The Public 
Service Commission then established a process to improve its reporting. Its 
Annual Report to Parliament for 2001-02 provides significantly better 
information than in previous years. It provides more explanation about the 
modernization of staffing practices in the public service, the health of the 
staffing system, and some of the human resources challenges facing the 
federal public service. The report is more candid and open about the 
problems of the current staffing system. 


5.54 However, the report refers to the Post-Secondary Recruitment program 
mainly as an example of the use of electronic recruitment, and it lists 
activities without discussing trends or the impact of the changes that have 
taken place. As an indicator of improvement in the program, for example, it 
gives only the increase in the number of applications received. It states that 
the 22,305 applications received during the fall 2001 campaign represent an 
80 percent increase over the previous year. But it does not assess the 
contribution of the Post-Secondary Recruitment program to maintaining and 
renewing an educated and skilled government workforce while reflecting 
public service values, such as making sure that the composition of the public 
service reflects Canada’s labour market. Nor does the report mention as a 
positive feature of the program that it meets or exceeds most of the 
government’s employment equity targets. 


§.55 The Commission’s annual report does not discuss the challenge of 
encouraging departments and regions to use the program more. Further, it 
discusses the Post-Secondary Recruitment program in isolation from 
government policies and other government initiatives that may influence the 
level of use of the program. For example, when the Commission delegates 
authority to departments for external recruitment, they use the program less. 
In 2001 the Department of Finance received delegated authority for a two- 
year trial to recruit into three streams of economists. Finance now conducts 
its own recruitment campaign, although it had been a regular user of the 
program before. 


5.56 Enhancements to the student bridging program make it a viable 
alternative to the Post-Secondary Recruitment program. Other, more general 
policies may affect the use of the program in the future—an example is the 
Treasury Board’s new policy of converting term employees to indeterminate 
status after three years of continuous government employment. 


5.57 Recommendation. The Public Service Commission should provide 
more information on the outcomes of the Post-Secondary Recruitment 
program and explain the program’s effectiveness relative to that of other 
recruitment initiatives and policies. 


Public Service Commission’s response. The Commission will seek the co- 
operation of the employer, departments, and agencies to describe in its annual 
report the results of its recruitment programs in the context of other policies 


and programs that also influence recruitment. 
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Conclusion 


5.58 In response to our recommendations of 2000, the recommendations of 
the Standing Committee on Public Accounts, and the Public Service 
Commission’s and Treasury Board Secretariat’s joint Recruitment Action 
Plan, the Public Service Commission has succeeded in redesigning the Post- 
Secondary Recruitment program to better meet departments’ renewal needs 
and to make it more attractive to university graduates. Qualified students are 
applying in large numbers and more are being hired than in previous years. 
However, it is too early to determine the program’s effectiveness in 
contributing to the overall renewal of the federal public service. 


§.59 Although the number of graduates hired through the program has 
increased, the program’s proportional contribution to new hiring from outside 
the public service has remained the same. In our opinion, the program’s 
effectiveness at hiring graduates as part of the renewal of the federal public 
service is limited by the lack of human resources planning and support for the 
program in departments. 


5.60 As the government’s main structured program for recruiting university 
graduates, the Post-Secondary Recruitment program should be a key part of 
public service renewal—although the government has yet to determine what 
will happen to centrally run programs under the new legislation, which 
proposes the delegating of staffing authority to departments. Federal 
departments face simultaneously the prospect of a high rate of retirement, 
particularly from their upper and middle ranks, and the challenge of 
recruiting people with higher education and levels of skill. 


§.61 Although the level of indeterminate hiring has increased in relation to 
term hiring over the last two years, managers still use term hiring for 
expediency despite the opportunity to hire for indeterminate positions from a 
pre-screened inventory. They still need to change their perspective on 
recruitment to strike a better balance between the immediate need to fill 
vacancies and the strategic needs of their departments. 


§.62 In co-operation with departments and agencies, the government needs 
to analyze its global requirements for recruitment and renewal. It also needs 
an accurate profile of the education and skills of people who are entering the 
public service through the various means of recruitment. Only then can it use 
its recruitment programs efficiently and promote more effectively the careers 
it can offer to university graduates. 


5.63 The Public Service Commission has improved the way it reports to 
Parliament on the results of its recruitment program activities. However, it 
needs to describe these results in the context of other policies Ue programs 
that also influence recruitment. 


Public Service Commission’s comment. The Public Service Commission 
accepts the findings, which recognize the progress achieved in program design 
since the Auditor General’s December 2000 Report. The Commission 
continues to improve the program and has implemented an extensive 


16 


Chapter 5 


Report of the Auditor General of Canada—May 2003 


POST-SECONDARY RECRUITMENT PROGRAM OF THE FEDERAL PUBLIC SERVICE 


marketing strategy to increase the use of the Post-Secondary Recruitment 
program by departments and agencies. On its Web site the Commission is also 
adding information on, and links to, departments and agencies that recruit 
for specific jobs. 


Treasury Board Secretariat’s comment. We concur with the majority of the 
findings and recommendations. With regard to the new Term Employment 
policy, we will be monitoring its implementation and will take steps to ensure 
that it does not have a negative impact on recruitment through the Post- 
Secondary Recruitment program. We believe that marketing the program and 
educating managers about it are the most effective ways to ensure that it is 
used fully. We will continue to work with departments and agencies and the 
Public Service Commission to identify human resources capacity shortages 
and develop strategies to respond to them. 
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About the Follow-up 


Objectives 

The objective of the follow-up was to determine whether changes to the Post-Secondary Recruitment program have 
made it more effective at hiring graduates as part of the renewal of the federal public service. 

Scope and approach 


We looked at whether the Public Service Commission met its commitment to redesign the program and improve the 
program’s marketing, and whether this had led to the desired results. We assessed whether the changes had affected 
the use of the development programs that come under the Post-Secondary Recruitment program. 


We interviewed Public Service Commission staff and human resources officials of six selected departments to explore 
reasons why departments are or are not using the Post-Secondary Recruitment program and to seek their views on 
the redesign. 


We looked at the Public Service Commission’s marketing of the program on university campuses and conducted 
interviews with some university staff to gauge the impact of the changes. 


Given that the redesign of the program is recent, we also looked at whether plans for further improvements were 
being developed. Finally, we looked at whether the Public Service Commission has improved the information it 
reports to Parliament on the program, including the outcomes. 

Audit team 

Assistant Auditor General: Maria Barrados 

Principal: Kathryn Elliott 

Esther Becker 


Denise Coudry-Batalla 


For information, please contact Communications at (613) 995-3708 or 1-888-761-5953 (toll-free). 
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Reform of Classification and Job : 
Evaluation in the Federal Public Service 


All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 
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Reform of Classification and Job 
Evaluation in the Federal Public 
Service 


Main Points 


6.1 The government has been trying since 1991 to reform the classification 
and job evaluation system in the federal public service in an effort to resolve 
long-standing problems. In May 2002 it announced that its second attempt at 
reform, the Universal Classification Standard, could not be implemented. In 
the work to convert positions from the old classification system to a new 
compensation structure using the new Standard, a single pay structure for all 
positions in the public service could not be achieved without causing 
unmanageable disruption to the federal workforce. The Treasury Board 
Secretariat could not find a way within the existing salary protection policy to 
reconcile the funds available for the conversion with a manageable rate of 
salary protection that management and unions would accept. 


6.2 Thus, in April 2001 the Secretariat decided not to implement the 
Standard government-wide. This planned universal approach has now been 
abandoned, despite a large investment of time and effort by tens of thousands 
of employees and an estimated investment of about $200 million in 
incremental costs between 1998 and 2001. 


6.3. The Secretariat has since embarked on a third attempt to reform 
classification. The new approach is a modification of the old classification 
system. Like the old system, it will use separate standards for each 
occupational group, making it difficult to compare the value of the work of 
different groups and related differences in pay. 


New issues 


6.4 The President of the Treasury Board announced in May 2002 that the 
government would begin developing a new step-by-step approach to 
classification reform, tailored to specific occupational groups. The Secretariat 
is committed to using some of the work completed under the Universal 
Classification Standard project as a basis for developing the new group- 
specific standards, thereby recouping part of the investment made in 
developing the Standard. Our follow-up found the Treasury Board Secretariat 
still at the planning stage of the new approach. 


6.5 The Secretariat will have to manage a number of challenges to succeed 
in this third attempt. We have concerns about the new approach: 


* The Secretariat has not defined an overall vision of what classification 
reform will entail and has not articulated a comprehensive plan and 
timetable for completing the reform. 
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« Through the new approach, the Secretariat aims to develop 
classification standards that meet the specific needs of each 
occupational group across the public service while also meeting the 
business needs of departments and agencies. However, the business 
needs of some 60 entities are varied and diverse. 


6.6 Some long-standing and intractable issues will need to be resolved: 


¢ There are no objectives and no formal policy for compensation in the 
public service to guide the new approach to classification and 
compensation reform. 


¢ There is still a need to align the current pay structures—mainly the 
products of separate negotiations with 16 unions over the past 
35 years—with new classification structures to be developed that are 
intended to reflect the value of the work employees are doing today. 
This alignment is needed to ensure that the reform will succeed this 
time. 


* The Secretariat is still developing a strategy for meeting the pay equity 
requirements of the Canadian Human Rights Act. 


6.7 Over the past 10 years, the Secretariat did not exercise sufficient 
control over the classification of positions in the public service to ensure that 
positions were classified accurately. Between 1993 and 1999, about 

28,000 promotions—almost one third of all promotions—were awarded 
through the reclassification of positions. The Secretariat does not know how 
many of these positions may have been overclassified. Since classification is 
directly linked with compensation, there is a risk that the government is 
overpaying some of its employees. 


6.8 In our interviews, we found that although there had been no formal 
post mortem, each senior official of the Secretariat we met with had learned 
lessons based on the scope of his or her responsibilities. A key change made 
was the reorganization of the Secretariat’s Human Resources Branch. 


The Treasury Board Secretariat has responded. The Secretariat has 
accepted two of our recommendations and indicates in its responses the 
action it intends to take. In its response to our third recommendation, it is 
not clear whether the Secretariat agrees or disagrees with the 
recommendation. 
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Introduction 


6.9 The reform of classification and job evaluation in the public service 
was one of the most important changes to come out of the November 1990 
White Paper, Public Service 2000: The Renewal of the Public Service of Canada- 
the initiative that was known as PS 2000. As part of service renewal, the 
government wanted to simplify employment and personnel management in 
order to redirect resources toward services to the public. 


6.10 A PS 2000 task force was set up in 1990 with a mandate to examine 
the design and administration of the classification and job evaluation systems, 
taking into account the values and objectives desired for a renewed public 
service. 


6.11. From 1991 until early 2001 the Treasury Board Secretariat, as the task 
force recommended, worked to reduce the number of occupational groups 
and simplify the practice and administration of job evaluation. The task force 
suggested developing a single job evaluation plan that would reflect the 
requirements of the Canadian Human Rights Act. 


Why robust classification and job evaluation systems are needed 


6.12 In the federal public service, the classification and job evaluation 
systems are considered the cornerstone of human resources management. 
Because collective bargaining is based on occupational groups, there are 
important links between classification and job evaluation and other aspects of 
human resources management such as staffing, organizational design, and 
bargaining unit structure. As of March 2002, the federal public service 
employed some 168,000 people and had a payroll of about $9 billion. In an 
organization of this size, any change to classification and job evaluation can 
have significant multiple effects on the rest of human resources management. 
(Our May 1996 Report, Chapter 5, Exhibit 5.4 describes some key links 
between classification and job evaluation and other aspects of human 
resources management.) 


6.13 What is more important, since job evaluation is primarily a way to 
establish internal equity in an organization, the evaluation results serve as a 
foundation for determining basic pay. Other variables such as the external 
market, collective bargaining, and how much the organization can afford to 
pay will also affect basic pay. Together, these factors contribute to ensuring 
that the organization provides equitable pay for all of its employees and that it 
can attract, retain, and motivate skilled employees. 


Why the classification and job evaluation systems needed reform 


6.14 In 1990, positions in the federal public service were classified into 

72 occupational groups and 106 sub-groups and into a number of levels in 
each of these groups. This occupational group structure had been created 
over 35 years ago, mainly as a basis for collective bargaining, and there were 
close to 80 bargaining units. As a result of this complex occupational group 
and bargaining structure, the public service had some 840 pay rates and 
70,000 rules governing pay and allowances when PS 2000 began. 
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6.15 In the federal public service, classification and job evaluation are 
intertwined. Classification assigns positions to the appropriate occupational 
group. Job evaluation determines the relative value of jobs, based on the 
content and requirements of each job. The evaluation results are then used to 
determine which positions in each occupational group should receive the 
same rates of pay, using a classification standard for each occupational group 
with a series of factors and a rating guide as well reference positions. 


6.16 As the task force pointed out in 1990, both the classification and the 
job evaluation systems were cumbersome, too complex, very time-consuming, 
and too costly to operate—to say nothing of their impact on costs in other 
areas of human resources management. The task force further recognized 
that the two systems presented many obstacles to the redeployment of 
resources and, in many cases, changes in technology and work organization 
had made them obsolete. They were highly susceptible to manipulation, in 
the view of the task force. It also found that the job evaluation system was not 
responsive to pay equity concerns. 


6.17. A long history of attempts to reform classification. In fall 1993 we 
audited the “final draft” of a new job evaluation system developed by the 
Treasury Board Secretariat, the Universal Classification Standard. We 
audited it before it was fully implemented, to see whether it would achieve 
the Secretariat’s objectives—one of which was to simplify the occupational 
group structure by reducing the 72 occupational groups to 23, as the PS 2000 
task force had recommended. We also wanted to examine a number of 
concerns that stakeholders had raised about the design and development of 
the Standard. 


6.18 We reported on that audit in our May 1996 Report, Chapter 5. We 
concluded that the Standard was a step in the right direction but would have 
to be improved substantially to be workable. We noted that the Treasury 
Board Secretariat had already acknowledged the Standard’s shortcomings 
and had begun to redesign it. 


6.19 In 1999, the government simplified the occupational group structure 
by reducing it from 72 to 29 occupational groups, with a new legal definition 
of each group. Asa result, however, not all occupational groups had a distinct 
classification standard. New occupational groups that had merged a number 
of former occupational groups continued to use the old groups’ classification 
standards and designations (such as CR for the clerical group and PM for the 
program management group, which are now within the program and 
administrative services or PA group). Twenty-five groups retained their union 
affiliations; the four others were exempted from union representation and 
collective bargaining. 


6.20 In 2000 we followed up on the development of the Standard, and in 
our December 2000 Report, Chapter 22, we concluded that significant issues 


still had to be addressed: 


* Before converting to the new Standard, the Secretariat needed to ensure 
that departments were applying it appropriately. 
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+ Departments needed to ensure that their work descriptions and position 
evaluation results were valid and reliable. 


¢ The Secretariat needed to test the Standard further to determine to 
what extent it was universal and gender-neutral. 


¢ The government needed to ensure that a reasonable and realistic level 
of financial resources would be available to convert positions to the new 


Standard. 


6.21 A change in direction: An end to the universal approach. In 

May 2002, the President of the Treasury Board declared that a universal 
approach to the classification system—applying a single classification 
standard and a single pay structure to all public service positions—was not 
workable. She announced that classification reform was moving ahead but in 
a substantially different direction—from a universal approach to one tailored 
to each occupational group. 


Focus of the follow-up 


6.22 The government has identified the importance of and need for 
classification reform for some time. We have audited it and will continue to 
give it attention. 


6.23 In our new approach to following up on previous audits, we re-audit 
the most significant areas—those that are systemic and current and that carry 
higher risk. In some cases, we extend our work to new issues that are related 
to the original audit. Normally, we would follow up.on our previous 
recommendations, but since the government has significantly changed its 
approach to classification reform, we focussed on its progress from 2000 to 
2002 and the challenges that remain. 


6.24 = Our follow-up looked at the continuing need for reform, the status of 
the new approach, lessons learned in the reform efforts to date, and 
corrective action identified and incorporated into planning the new 
approach. We also looked at how the Secretariat is ensuring in the meantime 
that it gives departments and agencies the clear direction and support they 
need to classify positions accurately under the old classification system. 


6.25 Our follow-up scope and criteria are set out in About the Follow-up at 
the end of the chapter. 


Observations and Recommendations 


The Standard is still recognized as a reasonable tool for job evaluation 


6.26 By the end of our first follow-up in 2000, the Treasury Board 
Secretariat had finished designing the Universal Classification Standard, 
except for confirming the final weighting of the elements and the number and 
point ranges of levels. In December 2000 we reported that the majority of 
departments we had consulted considered the Standard a promising tool for 


job evaluation. 
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6.27. A-key part of developing the Standard was establishing the relative 
importance of its elements to the objectives and values of the public service. 
In collaboration with all stakeholders, the Secretariat produced a reasonable 
ranking of job levels (from novice to expert) that reflected the values of the 
public service and incorporated the factors required by the Canadian Human 
Rights Act and Equal Wage Guidelines. 


6.28 In May 2001, the Canadian Human Rights Commission reported on 
the gender-neutrality of the Standard. It concluded that while some 
improvement was needed, the Standard was meeting the mark reasonably 
well. 


6.29 = In our current follow-up we found that the Secretariat had made 
significant progress on our December 2000 recommendation—to ensure that 
departments were implementing the Universal Classification Standard 
appropriately. For example, it intervened with departments in many targeted 
areas in an effort to make position evaluation results more consistent. It 
audited the quality of work descriptions as part of its quality assurance efforts 
and found that over 97 percent of the 966 work descriptions sampled were of 
medium to high quality. 


6.30 In consultation with departments, the Secretariat also put in place the 
Web-based Conversion Monitoring System to follow departments’ progress 
and determine how ready they were to convert their positions to the 
Standard. In July 2001, the Secretariat concluded that an average of 

77 percent of position evaluation results in departments were good enough to 
meet the requirements for conversion. However, departments still needed 
more time to reach the 95 percent expected by the Secretariat. 


6.31. During our current follow-up, senior officials of the Secretariat still 
asserted that the Standard could reasonably measure the full range of work in 
departments and was flexible enough to adapt to the evolving nature of work 
in the public service. They told us that the Standard is a useful tool. The 
Secretariat is committed to using some of the work already completed as a 
basis for developing new, group-specific standards and thereby recouping part 
of the investment made in developing the Standard. 


6.32 Secretariat officials noted a number of other residual benefits. Some 
organizations took advantage of the application of the Standard to streamline 
their organizational structures and move to more generic position 
descriptions, thus reducing administrative burden. 


A large investment in classification reform over the last 12 years 


6.33. More than 12 years have passed since the Public Service 2000 task 
force and the White Paper initiated the first attempt to reform classification 
in the public service. During those years, tens of thousands of employees from 
some 60 departments and agencies devoted significant amounts of time and 
energy to writing, evaluating, and revising work descriptions for some 
150,000 employees—more than once. Representatives of the 16 unions also 
participated actively in the many activities generated by the Universal 
Classification Standard project. 
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6.34 Since there was no mechanism in place to track the costs of this major 
project in departments or in the Secretariat, in 2001 the Secretariat asked a 
third party to design a costing methodology and to estimate the size of the 
departmental investments in the Standard project between 1998 and 2001. 
The study was to estimate only incremental costs, defined as “costs that 
would not have been incurred in the absence of the Universal Classification 
Standard project”—in other words, only the costs of additional salaries for 
new positions or of replacements for employees seconded to the project, 
overtime, infrastructure, training of managers and employees in the use of the 
new Standard, contracts, and system development directly related to the 
project. They did not include the ongoing costs of salaries for classification 
staff or for employees and managers involved in writing work descriptions or 
participating in evaluation committees. 


6.35 The study looked at five large departments, and in 2001 it reported on 
their incremental cost for the Standard project as a basis for estimating the 
incremental costs government-wide. The study estimated the total 
incremental cost of the Universal Classification Standard project at about 
$200 million from 1998-99 to 2000-01. The study report noted that while the 
estimate was not perfect, it was reliable enough to provide global indicators of 
incremental cost. We could not audit the estimate, but we reviewed the work 
it was based on and we believe it is a reasonable estimate of incremental costs; 
there is no good estimate of total costs for the 12 year period. According to 
the Secretariat, new funding accounted for only $25 million; other costs were 
absorbed by reallocations within departmental budgets. 


The development of a single pay structure could not be achieved 


6.36 Developing a universal approach to compensation that would fit a 
universal approach to classification represented a significant change in 
direction from the last 35 years. 


6.37 The PS 2000 task force assumed that with the implementation of the 
Standard, a common job evaluation plan would support the concept of “equal 
pay for work of equal value” and would rely on internal relativity as the 
underlying principle of compensation in the public service. The focus of the 
Treasury Board Secretariat’s Universal Classification Standard Division was 
on internal relativity as a basis for pay. 


6.38 To implement the Universal Classification Standard as planned, the 
Secretariat had to develop a single pay structure that the government could 
afford and that key stakeholders would accept, particularly senior 
management and unions. 


6.39 In developing such a pay structure, the Secretariat had to work with 
the following constraints: 


* Costs needed to be kept within the available funds. 


* The salary protection policy could not be changed (no employee was to 
suffer a financial loss as a result of the conversion). 


¢ Salary protection rates had to be acceptable. 
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¢ The rates of pay in the new pay structure resulting from the conversion 
had to be negotiated with unions. 


« The compensation solution could not increase the government's pay 
equity liability. 


6.40 From 2000 to 2001, the Secretariat worked to develop a single pay 
structure for all occupational groups (except the executive group) by 
consolidating the 72 existing pay structures. 


6.41 In 2001, based on data from the December 2000 results of applying the 
Standard, the conversion of all positions to a single pay structure was deemed 
unfeasible with the $400 million available to fund the conversion. The new 
pay structure could be achieved within approved funding only by assuming a 
salary protection rate of about 50 percent. This would mean that about 

50 percent of employees would have seen their classification levels lowered 
upon conversion to the Standard but not their pay. The classification levels of 
a significant number of employees would have been raised as well as the 
maximum pay rates. Furthermore, the difference between some pay levels was 
found to be not a meaningful enough incentive to take on more 
responsibilities. According to the Secretariat, the single pay structure would 
have been counterproductive and would have significantly affected the 
satisfaction and motivation of public service employees. 


6.42 Senior officials told us they were particularly concerned in 2001 about 
several other difficulties that a single pay structure presented. Given the 
market forces, there was a strong probability that many exceptions to the 
single pay structure would have to be made to compete with the outside 
labour market for certain skill sets that were in high demand. This would 
quickly move the public service away from a single pay structure. Other 
concerns were that a single pay structure might compromise the right of 
unions to bargain; it could lead to a larger union or a critical occupational 
group setting a pay structure that was not appropriate for others; and it could 
be seen as a constraint on the employee's freedom of association. In our 
opinion, these are concerns that existed from the beginning of the Universal 
Classification Standard project and should have been dealt with while the 
Standard was being developed. 


Collective bargaining, affordability, and market forces have shaped existing pay 
structures 


6.43 In our December 2000 Report we noted that the Treasury Board 
Secretariat had to work to find an acceptable approach to compensation that 
took into account pay equity requirements, internal relativity, market forces, 
and affordability—difficult challenges to reconcile. 


6.44 Since 1967, pay rates for unionized occupational groups have been 
established separately through collective bargaining for each group. Until 
recently, the process of establishing pay rates focussed on balancing 
affordability against parity with the outside labour market. Internal equity 
among all employees in the public service was not given the same emphasis. 
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6.45 The collective bargaining process for the 72 occupational groups, now 
in some 24 bargaining units, has produced 72 pay structures over time that 
each reflected the cumulative trade-offs in bargaining. Furthermore, the 
bargaining power of some unions, given their size and/or the critical skills they 
represented, had an impact on settlements. The established patterns of 
collective bargaining were among the factors that contributed to the difficulty 
of arriving at a single pay structure. 


6.46 Before the development of the Universal Classification Standard, the 
federal public service did not have a good basis for assessing the internal 
equity of its occupational groups. Once the Standard was applied, the 
Secretariat plotted departments’ position evaluation results according to the 
present salaries of the positions on a pay structure that fit the available 
funding. It found that the majority of positions were outside the desired pay 
structure, either higher or lower than the pay that was proposed. 


Decisions on external and internal relativity affected compensation 


6.47 Over the years, several decisions on compensation have had an impact 
on internal relativity among occupational groups and on parity with the 
outside labour market. These decisions include the following: 


* A freeze on salaries from 1991 to 1997, causing some salaries in the 
public service to fall behind those of comparable jobs in the private 
sector. 

* The settlement in 1999 by a Canadian Human Rights Tribunal of pay 
equity complaints for six occupational groups. 


* The payment of special allowances through collective bargaining to 
13 occupational groups to help attract and retain people with skills that 
were in high demand on the labour market—for example, computer 
scientists and engineers. Intended to be temporary, these allowances 
have in effect become part of the pay structure for these groups, whether 
or not the competitive demand has since disappeared. 


* Compensation adjustments negotiated by each bargaining unit between 
1997 and 2002 that led to significant variability in wage increases. 


No corporate policy on compensation 


6.48 To maintain a balance between affordability and internal and external 
equity, an organization generally needs a sound corporate policy on 
compensation for all occupational groups, including executives. A 
compensation policy is designed to support the organization's achievement of 
its strategic objectives for human resources management as well as its 
business objectives. 


6.49 The federal government has not had a formally approved 
compensation policy to guide its compensation decisions, and the impact can 
be seen in the many salary structures it has to deal with. Although the 
principles of a compensation policy were discussed over the years by various 
officials of the Treasury Board Secretariat, there was no formal framework in 
place to balance competing objectives and to review the impact of collective 
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bargaining decisions for one group on negotiations for the others. In some 
cases, compensation issues discussed in collective bargaining were assessed for 
their potential impact on the government’s pay equity liability or internal 
relativity. However, we saw no evidence of a concerted, structured approach 
within the Secretariat and with stakeholders during the Standard’s 
development that led to significant changes in its approach to collective 
bargaining that might have smoothed the path for a single pay structure, 
allowing for conversion to the Universal Classification Standard. 


6.50 Although the Secretariat has done considerable work recently to 
develop a compensation policy for the public service, no such policy has yet 
been finalized. 


6.51 Recommendation. The Treasury Board Secretariat should complete its 
development of a coherent corporate compensation policy. The Secretariat 
should monitor and evaluate the effects of the policy’s implementation on 
compensation. 


Treasury Board Secretariat’s response. The Human Resources Management 
Office (HRMO) is currently developing a comprehensive policy, which will 
assist in shaping the federal government’s approach to compensation, 
including the development of collective bargaining mandates. A discussion 
paper is expected to be released this spring with the intent of having a policy 
approved in autumn 2003. 


Federal public service compensation should serve, within an all overall 
human resources policy framework, to attract, retain, motivate, and renew 
the workforce required to deliver business results to Canadians. There are 
several factors that influence compensation, including the private sector 
labour markets, internal relativity among groups, macro-economic policy, 
social policy, relevant legislation, union leverage, public opinion, economic 
conditions, and the state of government finances. The policy will provide 
guidance on how to balance these factors and the risks associated with them. 
Indicators will be developed to monitor the implementation of the policy as it 
affects compensation outcomes. 


Unrealistic assumptions used in establishing costs 


6.52. The conversion of some 150,000 positions to a single pay structure had 
to be done with the funds that were available. Making solid cost estimates 
was critical. 


6.53 In 1991, the Secretariat committed to achieving cost neutrality in 
converting public service positions to the Universal Classification Standard. 
That is, it assumed that in the long run, the costs of salaries for positions 
converted to the new pay structure would be balanced by the savings— 
making the exercise cost-neutral. 


6.54 In 1997, the Secretariat produced a first estimate of the conversion 
costs. Since actual evaluation results were not available at the time, it based 
its estimate on previous experience with classification conversions and other 
historical data and arrived at an estimate of $400 million over four years as 
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the total cost of conversion. The Treasury Board gave it Preliminary Project 
Approval, even though this was a very rough estimate. 


6.55 We note that the $400 million cost estimate represented 1.5 percent of 
the payroll. However, our audit experience and the academic literature 
indicate that conversion costs might range around 3 percent of payroll—and 
more, when adjustments for pay equity are required. 


6.56 By 1999, the Secretariat recognized that given the changing nature of 
work in the public service, cost neutrality was not achievable; it was dropped 
as an objective. The Secretariat refined the cost estimate to $434 million over 
four years, assuming a salary protection rate of 40 percent. 


6.57 In January 2000, the Secretariat reviewed the funding required for the 
conversion, based on a sample of actual position evaluation results in 
departments using the Universal Classification Standard. This review too, 
had to assume a salary protection rate of 40 percent in order to stay within 
the approved funding limit. 


6.58 There was no support among deputy ministers for a salary protection 
rate of 40 percent. They were concerned about the disruptive impact on 
workforce morale at a time when competition for staff was starting to grow. 
Although the existing salary protection policy stipulated that no employees 
would lose income as a result of the conversion, employees with salary 
protection tend to feel that their work is undervalued; other employees may 
consider them overpaid. 


6.59 The Secretary of the Treasury Board asked officials to work with a 
salary protection rate of five percent instead, a rate that the deputy ministers 
did support. Further cost analyses suggested that the cost of converting 
150,000 positions using a salary protection rate of five percent would be about 
$1.1 billion over a number of years. 


6.60 In January 2001, based on several compensation scenarios using the 
December 2000 results of position evaluations in departments, the 
Secretariat found that to convert the 150,000 jobs and stay within the 
funding limit of $434 million, it would have to use a salary protection rate of 
about 50 percent to 60 percent. In February 2001, the Secretariat concluded 
that a single pay structure did not appear viable. It could not find a way 
within the current salary protection policy to reconcile available funding with 
a salary protection rate that was manageable and acceptable to management 
and unions. 


6.61 The Secretariat considered two other approaches to compensation. 
The first would use one classification standard with several pay structures, 
based on the position evaluation results using the Standard. However, the 
Secretariat concluded that this option would offer little defence against 
supporting differences in pay in the collective bargaining process or against 


pay equity complaints. 
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6.62. The second approach was essentially the same as the old approach to 
classification and compensation, that is, a different classification standard 
and pay structure for each occupational group. The Secretariat believed that 
this approach would be less costly and less disruptive than either the first 
approach or a single pay structure. It recognized, however, that this approach 
to compensation risked increasing the government’s pay equity liability. It 
noted that if the reform were substantial, it would support the reform and the 
initiative to modernize the management of human resources functions across 
the public service. If the majority of the public service were left untouched, 
the ability to support broader human resources reforms would be similarly 
limited. 


6.63 The Secretariat established a consultation and decision-making 
process to validate the analysis and conclusions of its staff. In February 2001, 
a committee of assistant deputy ministers was formed to assess the full 
potential implications of not implementing the Standard as originally 
envisioned. The committee examined several options, including putting a 
halt to all classification reform, and reviewed the two options developed by 
the Secretariat. 


6.64 The committee concluded that classification reform was still needed. It 
developed a proposal for a step-by-step, tailored approach to classification 
reform, beginning with the program administration (PA) group, to recoup 
some of the investment in the Standard project. The PA group embraced 
nine occupational groups with the oldest standards and over 45 percent of 
public service employees. It is represented in every department and includes 
positions from the lowest to the highest levels of the classification structure 
(excluding executive levels). Two of the former groups were involved in the 
major pay equity complaint against the federal government. 


6.65 In April 2001 the committee of assistant deputy ministers presented its 
recommendations to the Treasury Board Secretariat Advisory Committee (an 
executive committee of deputy ministers), which accepted the need to 
continue with classification reform and endorsed the tailored approach. 


6.66 Early in 2001, deputy ministers and some assistant deputy ministers 
were aware that the Universal Classification Standard project was unlikely to 
proceed as initially planned. However, they agreed to continue improving the 
quality of work descriptions and position evaluation results while new 
alternatives were explored and a new approach approved. 


6.67. From April 2001 through April 2002, the Secretariat engaged in 
intense education, consultation, and briefings of senior government officials, 
recognizing that these areas had not received enough attention during the 
project. It examined the feasibility of proceeding with the PA group, testing 
options with the largest departments. It worked on a submission to Treasury 
Board ministers that outlined options and its proposed new tailored approach. 


12 Chapter 6 Report of the Auditor General of Canada—May 2003 


Little communication for twelve 
months 


Report of the Auditor General of Canada—May 2003 


REFORM OF CLASSIFICATION AND JOB EVALUATION IN THE FEDERAL PUBLIC SERVICE 


No information was provided to stakeholders until May 2002 


6.68 From April 2001 to May 2002, the Secretariat made no new public 
announcements on the status of the Universal Classification Standard 
project—not on its Web site, in letters, or other announcements to 
departments. As of April 2001, the Secretariat no longer referred to the 
Universal Classification Standard project, and talked instead about 
“classification reform,” with no explanation. 


6.69 Secretariat officials told us that unions participated in some off-the- 
record discussions that explored solutions to implementing the universal 
approach, including what other options might be appropriate. However, in 
April 2001 and again in November 2001, the Public Service Alliance of 
Canada (PSAC) requested an update from the Secretariat on the status of the 
Universal Classification Standard project. According to PSAC records, in 
each case the Secretariat responded only that work was continuing. 


6.70 Following consultations with executives across the country in 

summer 2002, the Association of Public Executives (APEX) reported that 
public service executives showed a “definite hint of disengagement” from 
major reforms of human resource management and linked it directly to the 
failure to proceed with the Universal Classification Standard. Executives 
noted that they had invested considerable time and money and their personal 
reputations in projects that had not materialized. 


6.71 Furthermore, the performance information reported to Parliament on 
the Universal Classification Standard project did not adequately reflect the 
evolving state of classification reform. In its reports on plans and priorities for 
1998-99, 1999-2000, and 2000-01, the Secretariat said that it planned to 
implement the Universal Classification Standard (Exhibit 6.1). In its 1998-99 
Departmental Performance Report, it said it would implement the Universal 
Classification Standard as the cornerstone of other reforms. The Secretariat 
said also that conversion of the public service would be administered on a 
staggered basis throughout 1999. In its departmental performance report for 
1999-2000, it said it was implementing the Universal Classification Standard 
when in fact it was still testing the Standard’s application, and indicated that 
conversion would not be completed until 2000-01. However, its departmental 
performance report for 2000-01 dropped all direct reference to the Universal 
Classification Standard. Instead, the Secretariat began referring to 
“classification modernization.” There was no explanation for why the 
terminology had changed and what it meant, or its implications for human 
resources management modernization. 


6.72 Officials of the Secretariat told us that the announcement of a new 
approach was delayed for a number of reasons. They wanted to develop an 
alternative to classification reform, and that would require the approval of 
Treasury Board ministers before it could be announced. A tense labour 
relations climate, the events of September 11, 2001, and changes in the 
Secretariat’s senior management also contributed to the delay. 
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Exhibit 6.1 The Treasury Board Secretariat’s reporting on classification reform 


What the Secretariat planned 
in Report on Plans and Priorities 


1998-99 


Implement the Universal Classification 
Standard (UCS), a modern, universal and 
gender-neutral job evaluation tool that 
serves as the critical underpinning of 
other human resources management 
reforms in collective bargaining, 
compensation and staffing. 


What the Secretariat reported 
in Departmental Performance Report 


1998-99 


The Universal Classification Standard will 
serve as the cornerstone of human 
resources management reforms including 
collective bargaining, compensation and 
staffing...the government published the 
new Universal Classification standard 
occupational group definitions...reducing 
the number...from 72 to 29, thereby 
allowing for more effective human 
resource management and greater 
employee mobility. Conversion of the 
Public Service will be administered on a 
staggered basis throughout 1999 as 
collective agreements expire. 


1999-2000 


Implement the Universal Classification 
Standard 


Use the potential of Universal 
Classification Standard, post-conversion, 
to facilitate human resource management 
reforms in collective bargaining, 
compensation, staffing and organizational 
development. Negotiate the new 
compensation structure for the 
implementation of Universal 
Classification Standard. 


1999-2000 
Implementation of the new gender- 


| neutral occupational group structure—the 


Universal Classification Standard 


| (UCS)—continued during 1999-2000. 


Following some delays, the conversion is 
expected to be completed during the 
2000-01 fiscal year. Gender neutrality, 
simplicity and universality in job 
classification are the goals of the new 
standard. UCS is also an integral part of a 
modern human resources framework that 
will make the public service more flexible 
in serving Canadians in the 21°* century. 
Further details on the progress of 
implementation are available on the 
Secretariat's Web site. 


2000-01 

Begin using the Universal Classification 
Standard as a key corporate tool to 
stimulate change 


2000-01 


Work continued on preparing for 
classification modernization 


2001-02 
Modernization of the classification plan 


2001-02 

The Secretariat...implemented a number 
of reforms that...addressed challenging 
issues Such as...the improvement of the 
classification system... The Secretariat 
has identified viable options for a tailored 
classification reform program. It will be 
flexible, gender neutral, and conducive to 
recruiting and retaining the diverse 
workforce needed in the Public Service. 


Source: Treasury Board Secretariat Reports on Plans and Priorities and Departmental Performance Report 
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Old classification standards are obsolete 


6.73. One of the Secretariat’s responsibilities is to establish adequate 
classification standards and the controls to ensure that positions are classified 
according to those standards. We expected that given that responsibility, it 
would take steps to ensure that positions would be classified adequately 
throughout the development of the Universal Classification Standard and 
until the Standard was fully in place. 


6.74 Thirteen years after the PS 2000 classification task force made its 
recommendations, departments are still using the 72 old classification 
standards. Numerous problems with these standards have been identified 
over the years: 


* Many of the old standards do not include the four factors listed in the 
Canadian Human Rights Act and Equal Wage Guidelines and therefore 
cannot be used for analysis of equal pay for work of equal value. 


¢ Some are obsolete. 


* Some refer to work no longer performed and equipment no longer used 
in the public service. 


* Some exclude important dimensions of work today and do not 
distinguish meaningfully between degrees of job complexity. 


* Some classification standards do not measure unique aspects of work 
and consequently credit the same job content more than once. For 
instance, in some standards we find that the larger the budget to be 
managed or the higher the number of people to be supervised, the higher 
the value of the position. Since the budget to be managed increases with 
the number of people supervised, in effect the same aspect of work is 
credited twice. Furthermore, these classification standards support 
values no longer relevant to achieving efficient delivery of services and 
programs. 


No mechanisms for monitoring or controlling classification 


6.75 Departments have been delegated the authority to classify and also 
reclassify positions. A reclassification is the appointment of an employee, 
without competition, to a position whose occupational group, subgroup, or 
level has changed as the result of the position’s re-evaluation. 


6.76 Leading up to the planned conversion to the Standard, the Treasury 
Board Secretariat suspended its central monitoring of classification and 
conducted its last audit of classification in 1991-92. In the years when the 
Secretariat still controlled classification, about six percent of public service 
positions were either overclassified or underclassified. 


6.77 Reclassifications as a percentage of total promotions have remained 
relatively constant—around 30 percent—over the past 10 years, according to 
the Public Service Commission’s records. From 1993 to 1999, the Treasury 
Board Secretariat has had no mechanisms for controlling or monitoring 
classification. It thus has had no way of knowing whether any of the roughly 
28,000 positions whose reclassification resulted in promotions were 
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misclassified or the magnitude of the cost of any such misclassifications 
(Exhibit 6.2). In 1990, we reported that the Secretariat had estimated that a 
6.4 percent rate of misclassification in 1988-89 resulted in an unwarranted 
annual net cost of $5.3 million. 


Exhibit 6.2 Reclassifications resulting in promotions, 1993 to 1999 


Promotions 
25,000 23,744 
HB Without competition 
20,000 indeterminate reclassification 
le MH) Total indeterminate 
15,000 14,496 promotions with and 
without competition 
10,837 
10,000 7932 8,925 
6,694 
4,740 5,102 
9,000 | oz 3372 3,362 


1993-1994 1994-1995 1995-1996 1996-1997 1997-1998 1998-1999 
Source: PSC annual reports - Not audited 


6.78 In our 2000 follow-up we found that when the Secretariat began to 
apply the Universal Classification Standard in 1998, some departments and 
agencies had not reviewed the old classifications of their positions to 
determine whether they were accurate and up-to-date. We noted that no one 
had any idea how many positions at that time had been misclassified. 


6.79 In fall 2002, the Secretariat conducted a survey to capture an accurate 
overview of classification practices and capabilities across the public service. 
Preliminary results showed that nearly 35 percent of positions classified under 
the old standards over the past few years had been evaluated using the work 
description elements of the Universal Classification Standard, but without 
guidance from the Secretariat. The Secretariat recognizes that in the absence 
of appropriate guidelines, departments may have evaluated the Standard- 
style work descriptions against old standards in different ways and arrived at 
different results. 


6.80 Recommendation. The Treasury Board Secretariat should ensure that 
timely audits of classification are conducted and take all the necessary 
measures to restore and sustain the integrity of the classification of positions 
in the public service. 


Treasury Board Secretariat’s response. The Secretariat agrees on the 
importance of ensuring the integrity of the classification system. In line with 
the Treasury Board Policy on Active Monitoring, the Secretariat uses the 
term “active monitoring” rather than “audit.” This approach is well suited to 
the classification system because deputy heads are delegated the authority by 
the Treasury Board to make classification decisions. Departments are 
responsible for monitoring how positions are classified within their 
organizations and the Treasury Board Secretariat is responsible for supporting 
departments and monitoring the overall situation across all departments. 
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The Secretariat has reminded deputy heads of their responsibility for 
systematic monitoring to ensure integrity of classification in the public 
service. The Secretariat is implementing its active monitoring program in 
2003-04 on two levels. At the departmental level, it will assist classification 
managers to build their own active monitoring program or to align their 
current program with the active monitoring approach. At the corporate level, 
it will undertake a comparative study of how a classification standard that is 
commonly used by departments is being applied. Working with a selected 
department, the Secretariat will also test its active monitoring methodology 
and report on the results. New electronic, Web-based tools are under 
development to assist departments in the consistent application of the 
existing standards. The Secretariat will continue to advise departments on 
applying the classification standards and will deal with grievances and dispute 
resolution. All classification monitoring initiatives will be synchronized with 
other Secretariat review mechanisms to ease the burden on departments. 


The Secretariat is developing tools to work with the old system 


6.81 The Secretariat is working to make old classification standards usable; 
to bring all standards into conformity with the occupational group structure 
at a minimum; and to maximize the use of the work descriptions developed 
under the now ended Universal Classification Standard project. Documents 
from the Secretariat indicate that all these activities are taking longer than 
originally forecast. 


6.82 Over the last year, the Secretariat has started developing a number of 
measures to provide departments with direction and support for classifying 
positions using the old classification standards. 


6.83 The demise of the Universal Classification Standard project has 
increased the demand for reviewing classification levels using the old 
standards, much of which was deferred because the new Standard was soon to 
be implemented. Hard copies of the old standards are in short supply across 
the public service. At the time of our follow-up, the Secretariat was scanning 
its 72 old classification standards into electronic format and had informed 
departments that it would post them on its Web site early in 2003. 


6.84 The Secretariat is developing some monitoring tools for its own use 
and for use by departments, such as databases and report-generating tools 
that would allow departments to do peer reviews, update their data, and 
improve job relativity ratings among departments. The Secretariat is 
developing a classification monitoring framework that, in line with the 
Treasury Board policy on active monitoring, allows departments to monitor 
their management practices and controls. It expects to implement these 
initiatives in 2002-03 and to be actively monitoring the overall state of 
classification by 2003-04. 


6.85 As noted earlier, during the Universal Classification Standard project 
the majority of departments rewrote their work descriptions in the style of the 
Standard. The Secretariat is developing a tool to match the work described in 
the Standard-style work description format to the requirements of the old 
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classification standards. The tool is designed to assist departments in 
collecting information that may be missing from Standard-style work 
descriptions (such as number of people supervised, or size of budget) but that 
is needed to evaluate positions by the old standards. It plans to provide 
guidelines for evaluating the new work descriptions against old standards in a 
consistent way to ensure relativity throughout the public service. 


6.86 In our opinion, besides the other problems we have set out that are 
inherent in the old standards, they will not value or assess much of the job 
content captured in the Standard-style work descriptions. 


The end of the universal approach was announced 


6.87 On8 May 2002, the President of the Treasury Board announced that 
while classification reform was still a priority, a universal approach to 
classification was not workable in the current environment. The government 
would begin a new program of step-by-step, targeted, and tailored reforms. 
Exhibit 6.3 summarizes the key features of this new approach. 


6.88 The new approach will begin with occupational groups whose 
standards require fundamental changes. As of March 2003, the Treasury 
Board Secretariat had not finalized the identification of those groups or 
determined which classification standards would require only updates or no 
change. In the interim, the Secretariat has given priority to the rebuilding of 
capacity in departments and agencies to classify their positions using the 

72 old classification standards. 


Exhibit 6.3 A new direction for classification reform 


Key aspects of the new approach include the following: 
Standards will be tailored to respond to the specific needs of each occupational group. 


The government will continue to subscribe to the pay equity principle and awaits a 
proposal from the Pay Equity Task Force for an improved approach to achieving and 
maintaining pay equity. 


Tailored standards will reduce the need for the large number of standards currently 
used. 


Reforms will be carried out only for occupational groups that require major changes to 
their standards. 


Classification reform will be implemented on a three-year rolling plan. 
Classification reform will be carried out in co-operation with departments and unions. 


The reform of selected occupational groups will use, where possible, work already done 
under the Universal Classification Standard project such as the work description 
format. 


The new approach will support the modernization of human resources management, 
an initiative now under way. 


Source: Treasury Board Secretariat http://publiservice. tbs-sct.gc.ca/media/nr-cp/2002/0508 _e.asp 
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The new tailored approach faces significant challenges 


6.89 Classification reform is part of the government's initiative to modernize 
human resources management in the public service. However, it is not clear 
how classification reform will be aligned with the broader modernization 
initiative and its reform strategy and objectives. Further, the government has 
defined no clear vision of the overall results it expects to achieve with its 
incremental and tailored approach. Nor has it established a timeline for 
managing its classification reform. 


6.90 Classification is an essential component supporting many human 
resources management policies and systems in the public service. Therefore, 
it is particularly important to ensure that any reform of classification be based 
on a solid foundation. During our audit, the Secretariat told us that it was at 
an early stage of developing a framework for human resources management. 


6.91 The Secretariat has made a deliberate decision to correct problems as 
the reform of classification moves forward. This represents significant risks 
when the overall end results are not defined. For instance, a new 
classification standard to meet the specific needs of one group could be 
implemented without knowing the potential impact on other occupational 
groups slated for attention later. 


6.92 Similarly, the Secretariat intends to meet the specific needs of 
occupational groups across the public service while still responding to the 
business needs of departments. The mandate, organizational structure, 
programs, and services of some 60 departments and agencies are unique; 
attempting to reconcile the very diverse needs of all these organizations while 
developing a government-wide classification standard for each occupational 
group will be a significant challenge. 


6.93 One concern for the Secretariat is aligning the new classification 
structure with a new pay structure for each of the occupational groups 
selected for reform. In our opinion, since the constraints and conditions that 
hampered the universal approach still exist, the Secretariat faces the same 
challenges it encountered in the conversion to the Universal Classification 
Standard, albeit on a smaller scale. The biggest challenge will be to develop a 
classification standard for the PA group, a high priority and one of the three 
groups identified as candidates for the first wave of reform. The Secretariat is 
still assessing whether the PA group can feasibly be one of the first groups 
converted to a new standard, and it has indicated that conversion is some 
years ahead. 


6.94 Further, the tailored approach to classification may not address the 
Canadian Human Rights Act requirements. The new approach continues the 
old system’s use of separate standards for each occupational group, which 
makes it difficult to compare the work of different groups and thereby to 
justify differences in pay. The new approach risks incurring pay equity 
complaints and the associated potential liability. The Treasury Board 
Secretariat has recognized this risk and is developing a pay equity risk 
management strategy as part of its larger risk management framework in its 
Human Resources Branch. 
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6.95 Moreover, the tailored, group-specific classification standards will have 
to be gender-neutral to comply with the Canadian Human Rights Act and 
Equal Wage Guidelines. The Secretariat intends to use the Act’s 

four provisions (skill, effort, responsibility, and working conditions) in order 
to recoup part of the work completed in the Standard project. It recognizes 
that modifying any part of the Standard without ensuring that appropriate 
safeguards are in place for gender neutrality could increase the risk of pay 
equity complaints. 


6.96 Finally, the long-term plan and funding for classification reform are still 
uncertain. In February 2002 the Secretariat projected the costs over 14 years 
to reform classification standards. Given that the scope of classification 
reform has not yet been decided, it is not clear if this projection covers the full 
costs. Nor is there a clear end date or target for completing the reform. The 
Treasury Board Secretariat is working on a three-year rolling plan that 
requires it to return to Treasury Board ministers annually to review the 
progress and the costs of the reform. 


6.97 In our follow-up audit we expected to find that the Secretariat had 
conducted a comprehensive post mortem to determine the root causes of the 
reform project’s demise over the last 12 years and certainly over the course of 
the Standard project since 1995. To be meaningful, a post mortem would 
include all key stakeholders and ideally, after the second failed attempt at 
reform, would be guided by an independent group. 


6.98 In our interviews we found that although there had been no formal 
post mortem, each senior official of the Secretariat had learned lessons based 
on the scope of their responsibilities and their interpretation of the reasons 
for the failure. At the end of our audit, the Secretariat gave us a paper 
summarizing the views of its officials and indicating how they had based the 
new approach on the lessons learned. 


6.99 A key change was the reorganization of the Secretariat’s Human 
Resources Branch. During the second attempt to reform classification (1995 
to 2001), the Treasury Board Secretariat’s Universal Classification Standard 
Division was responsible for delivering the Standard as a job evaluation tool 
and ensuring that departments used it to get reliable results in position 
evaluations. While consultations happened on some issues, the divisions with 
related responsibilities (Classification, Labour Relations and Compensation) 
largely worked independently. The Secretariat has recognized the need for 
timely and structured sharing of information that is relevant to different 
divisions of the Branch and to use it as a basis for decisions. It has clustered 
compensation, labour relations, classification, and pay equity together for a 
more integrated approach to these areas. In our opinion, this reorganization 
should help the Secretariat deal with some of the intractable issues that it 
must resolve in order for reform to proceed. 


6.100 The Secretariat recognizes that classification reform is not in itself a 
solution to all problems of human resources management. It also recognizes 
the need to involve senior leaders in analyzing the problems and developing 
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solutions so the reforms will meet the business needs of departments. The 
Secretariat has committed to working with the public service unions. 


6.101 The Secretariat has said that it intends to strengthen the human 
resources management capabilities of departments, managers, and unions. It 
also plans to consult with organizations and experts with similar experience 
outside the Human Resources Branch and the federal public service. 


6.102 However, it is not clear to us how the new approach the Secretariat is 
taking will address the fundamental problems identified by the PS 2000 Task 
Force, problems that still exist today: the system is complex, cumbersome, 
very time-consuming, not responsive to pay equity concerns, and costly to 
operate. The Secretariat has projected the costs of classification reform to 
2014 and has announced a strategy and a three-year rolling plan to be revised 
yearly. But there are no overall objectives for reform, no global plan, and no 
specific target dates for achieving results. In our opinion, it is questionable 
whether a reform initiative staggered over many years can be effective. 


6.103 Recommendation. The Treasury Board Secretariat should articulate its 
objectives and expected outcomes for classification reform and develop a global 
plan for addressing the root problems inherent in the old classification system. 


Treasury Board Secretariat’s response. The government's goal is to simplify 
and modernize job classification by introducing a balanced system that 
reflects labour market realities, treats men and women equitably, and helps it 
recruit and retain the people needed to serve Canadians. In May 2002, 

the Treasury Board announced its decision to implement a multi-year 
classification. Subsequently, implementation initiatives were identified and 
linked into a coherent three-pronged strategy: 


* development of new, tailored, gender-neutral standards responsive to 
the specific needs of individual occupational groups, where existing 
standards are particularly outdated; 


* ongoing maintenance of existing standards, where necessary, and 
guidance on their application; and 


- rebuilding of system capability through active monitoring and the 
development and delivery of an up-to-date curriculum for technical 
specialists and line managers. 


Through the adoption of a three-year rolling plan, updated annually, the 
Treasury Board Secretariat will focus on priorities agreed with departments 
and unions. Setting business-based priorities, pacing change to fit the system’s 
capacity to manage reform, and resourcing appropriately are the best means 
to produce sustainable results. 


Work to achieve these aims is well under way. Several departments, public 
service unions, and functional communities have presented their business 
case proposals for classification reform. The decision as to which occupational 
groups will be selected for inclusion in the first formal three-year rolling plan 
will be announced this spring. During 2003—04 we will also be launching our 
active monitoring program and the first phase of our updated training 
curriculum. 
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Conclusion 


6.104 Despite 12 years of effort, the government has not made significant 
progress in reforming its old and problematic classification system. The 
Treasury Board Secretariat changed its direction on classification reform 

in 2001; it realized it could not develop a manageable single pay structure 
with the available funding. In our view, several factors contributed to the 
failure of a universal approach: the lack of a formal compensation policy; the 
pattern of collective bargaining over the last 35 years to determine pay; the 
lack of co-ordination among classification, compensation, labour relations, 
and pay equity; and flawed costing assumptions. While the Secretariat has 
reorganized to better co-ordinate its human resources activities, many of the 
same factors remain to challenge the new approach. 


6.105 The Treasury Board Secretariat finished designing the Universal 
Classification Standard except for confirming the final weighting of the 
elements and the number and point ranges of levels. The Secretariat says it 
will use the work done as a basis for developing new group-specific standards. 


6.106 The current state of classification standards, some of them more than 
30 years old, is a questionable foundation for a new and coherent 
compensation plan across the public service. The government's ability to 
justify the adequacy of classification and therefore of compensation in the 
public service is vulnerable, after a long period with little central control over 
classification practices in the public service. 


6.107 The extent to which positions in the public service today are 
misclassified is unknown. However, given the number of positions reclassified 
in the last decade when the Secretariat was not controlling or monitoring 
classification actions, we view the state of job classification and compensation 
with serious concern. The Secretariat will need to rebuild and maintain the 
credibility of the old system until it moves to something else. 


6.108 The new, tailored, incremental approach to classification reform that 
the Secretariat is planning faces significant challenges. It is not clear how it 
will address pay equity concerns and support modernization in other parts of 
human resources management—that are founded on the classification 
system. The government needs a more comprehensive plan to address these 
challenges. 


Treasury Board Secretariat’s comment. The Treasury Board Secretariat 
agrees on the importance of modernizing classification in the public service. 
This must be done in a way that helps us attract and retain the talent we need 
to serve Canadians and that treats men and women equitably. 


It is evident that we made a substantial investment of time and energy in 
pursuit of a “universal” approach to classification that proved unworkable. 
We regret that this much-anticipated project did not fully achieve the 
intended result. 
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Much of the work done to prepare for the planned “universal classification 
standard” (UCS) has proved to be a worthwhile investment. In the main, 
existing resources were used by departments to carry out day-to-day 
classification activities. Many employee work descriptions were significantly 
updated. The strategic use of generic job descriptions has greatly reduced the 
work required to manage the personnel system. We have introduced 
innovative gender-neutral approaches into our classification standards design, 
work description writing, and job evaluation procedures. 


Since the Treasury Board President’s announcement in May 2002 of a group- 
by-group approach to classification reform, we have made solid progress: 


* We have consulted widely with departments, unions, and functional 
communities on 18 groups proposed for priority attention. In May 2003, 
the President will confirm our initial priorities for classification reform. 


> The 72 current classification standards have been mapped against the 
UCS-based work descriptions to ensure public-service-wide consistency 
of interpretation and application. This spring, we will make all job 
evaluation standards accessible through the Internet. 


¢ Last autumn we surveyed departments on classification capacity and 
current practices. The results are assisting us to focus curriculum 
development, training, and the sharing of best practices. In June, we will 
hold the first interdepartmental symposium of classification professionals 
to promote the renewal of the community. 


We are convinced that our reporting to Parliament was appropriate. We could 
not announce a change of direction before a decision by Treasury Board 
ministers. The analysis leading to this decision did take longer than we would 
have liked, for the reasons stated in paragraph 6.72. In the meantime, 
however, senior Treasury Board Secretariat officials did appear before the 
Public Accounts Committee and indicated that we were exploring alternative 
approaches to classification reform. 


In respect of pay equity, the departments of Justice and Labour and the 
Canadian Human Rights Commission recognize that the current provisions of 
the Canadian Human Rights Act are difficult to interpret and apply. For these 
reasons, the Bilson Task Force, appointed in June 2001 by the ministers of 
Justice and Labour to review pay equity legislation, is studying these 
provisions now. 


Because the recommendations of the Task Force could result in changes to 
the current policy or legislation, we must take this into account in 
determining how to best balance our responsibilities to respect the provisions 
of the Canadian Human Rights Act, the Financial Administration Act and the 
Public Service Staff Relations Act. We have undertaken several initiatives to 
better manage our pay equity obligations, such as working at enhancing our 
capacity to monitor the impact of changes in gender predominance on wages. 


In summary, we are trying very hard not to repeat the mistakes of the past as 
we manage a very complex issue. 
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About the Follow-up 


Objectives 


The objective of this follow-up was to assess the progress in job classification reform in the federal public service 
since 1996 and to identify the challenges ahead. 


Scope and approach 


The reform of job classification in the federal public service started in 1990 in the context of Public Service 2000 
(PS 2000). We situated the current follow-up in the context of the principles, objectives, and recommendations 
submitted by the PS 2000 Task Force, the objectives of the Universal Classification Standard (UCS) project in 1995, 
and the progress made until December 2000. 


The primary focus of our work was on the initiatives undertaken by the Treasury Board Secretariat in the 
implementation of the Universal Classification Standard between December 2000 and December 2002. We 
examined the work and rationale leading to decisions on changes to the Universal Classification Standard project. 
We reviewed relevant documentation and conducted many interviews to gain a clear definition of the new approach 
to classification reform announced by the Secretariat in May 2002. We reviewed the analyses conducted by the 
Secretariat during the decision-making process. We also examined the extent to which the Secretariat had taken 
measures to identify the key factors that would ensure the success of future classification reform. Finally, we looked 
at how the Secretariat is ensuring in the interim that departments and agencies have clear direction and the 
necessary support to classify positions adequately under the old classification system. Our follow-up excluded audit 
work in departments and agencies. 


Criteria 
The criteria are drawn from best practices in the areas of classification and job evaluation, risk analysis management, 
and decision-making processes used in previous studies and audits of the Office. We expected that 


* decisions on changes to the classification system would be supported by sound analysis, using all relevant and 
up-to-date information including the position of key stakeholders; 


* key critical success factors for future classification reform would have been identified, based on lessons learned; 
and 


* in the interim, departments and agencies would have clear direction and the necessary support to classify 
positions adequately. 
Audit team 
Assistant Auditor General: Maria Barrados 
Principal: Kathryn Elliott 
Director: Ghislaine Cété 


Robyn Roy 
Marie-Eve Viau 


For information, please contact Communications at (613) 995-3708 or 1-888-761-5953 (toll-free). 
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